Chia plotting basics
— February 22, 2021
Introduction
First it is important to know that there are two very different parts of being a Chia farmer. There is creating the plots or plotting and then there is farming the plots. In this post we are going to focus on the process of creating your plots. The types of machines and storage space are very different than the types of hardware you ultimately want to use to farm. You can see some example farming rigs on our very useful repository wiki.
We initially recommend that you try plotting with what you have around. The only caution about that is that you want to limit the amount of plots you create that use your internal/consumer grade SSD as the temporary space. SSDs have very different wear lives and we have detailed information on SSD endurance.
You really never need to plot a plot with a k size larger than 32. Those who do plot larger are either doing them to show off (and we encourage this for fun) or to optimally fill the open space on a specific drive. A k32 will take up 101.3 GiB of space once completed but will need a total of 239 GiB of temporary space as it is being created. A single k32 plotting process never needs more than 239 GiB of space. One needs to be careful here as 239 gibibytes uses 1024 as its divisor where GB or gigabytes uses 1000 as the divisor. That means you will need 256.6 GB of temporary space and the final plot file will take 108.8 GB. A k32 plot can be done by one expert we know in just under 4 hours, but most experts are creating plots in 5 hours and most folks average around 9-12 hours.
Creating a plot is a process that will take RAM, CPU cycles, IO to your disks and it will use them differently in each of the four phases of plotting. Everyone wants a magic “right” answer or to use AI to figure out the optimal plotting strategy for their machine. However almost every machine is different along one of these parameters so you just have to try. Longer term we will be able query your machine and make some recommendations but that is not today. You really will have to test. And no, the experts in the various Keybase channels don’t know your best settings either.
Getting going
The first phase generates all of your proofs of space by creating seven tables of cryptographic hashes and saving them to your temporary directory. Phase 2 back-propagates through the hashes, phase 3 sorts and algorithmically compress these hashes in the temporary directory while starting to build the final file and phase 4 completes the file and moves it into your final plot destination.
One of the major bottlenecks is usually the total sustained write speed of the disk underneath your temporary directory. We recommend used datacenter SSD if you really want to go fast and not sacrifice consumer SSDs making plots. NVMe is faster than SAS and SAS is faster than SATA. This PC World overview of storage technologies can explain these acronyms and the differences. TBW, or terabytes written, is generally how SSD drive life is measured. One k32 writes 1.8TiB in non-bitfield mode and 1.6 TiB with bitfield enabled. More on bitfield in a moment.
Making the single fastest plot isn’t generally the best plotting strategy however. Often you’re getting amazing speed because you’re using the turbo core of that multi core processor. The folks who plot the most have shown that you should measure in TB (TiB if you’re old school like us) per day. The way to get the maximum TB/day is to plot lots of plots in parallel. Some of the top plotters use datacenter SSDs. Some use SAS drives. Raid 0 is often very handy to tie together a couple of small fast drives into one say 2TB partition so you could fit 5 k32 temporary spaces on that one virtual RAID drive.
All of that said, for my personal plotting I use a 2017 iMac and a 12TB Western Digital external drive on USB 3.0 for both temporary and final directory, and I get a k32 about every 10 hours.
Good assumptions
There are some good rules of thumb for now. These can change as we will be returning to making some plotting speed improvements after launch. First we need to explain bitfield versus no bitfield plotting. Originally, the plotter did not use bitfield back sorting. The bitfield back sort is theoretically faster than not using the bitfield and we already know that it saves 12% of total writes but requires more RAM. We have a hunch we can speed bitfield up 10% and make it work on more processors but that’s not in there yet. What we do know is that, as long as you’re ok with the 12% more total writes, no bitfield will work faster when SSD or fast SAS is your temporary directory. If your temporary directory is on a regular HDD, like mine is, bitfield is 20% faster than no bitfield. Older CPUs may not see the speed increase as much as noted above.
Returning to the rules, here are a few. Never touch the stripe size of 65536. No one has found a speed up over that value and we are likely removing it from the options list. (Update: as of 3/11/21 stripe size has been removed as an option.) You almost never want to use any bucket values other than 128. Less buckets requires more RAM for each plotting process. 64 buckets requires twice the RAM.
As far as number of threads are concerned you are generally going to want 2 to 4. More than 4 seems to have diminishing returns and 2 threads is a lot better than 1. More threads also require a bit more memory to successfully complete a plot. The threading is only used in phase 1 currently.
As of Chia 1.0.4, RAM requirements are almost identical between bitfield and no bitfield. This is a chart of the various RAM choices assuming a k32 with 128 buckets and 2 to 4 threads:
| RAM MiB: | Minimum | Medium | Maximum |
|---|---|---|---|
| Bitfield | 900 | 2640 | 3400 |
| No Bitfield | 900 | 2640 | 3400 |
Below minimum your plot will fail. Medium is enough RAM that you’ll get most speed improvements, but not all. This is useful when you’re trying to get more plotting processes parallel and have limited RAM. Using anything over the maximum is wasting RAM as you will not plot any faster. We are pretty certain of the minimums and maximums but there is community debate about the medium values. We’ll update this chart accordingly as we have better data.
Mastering plotting
Most people start plotting from the GUI. You can successfully complete a couple of plots in parallel from there to get the hang of things. As people choose to get more serious they migrate to the command line. It is worth noting that Windows suffers 5-10% slower plot times versus MacOS or Linux for now.
Once you get some experience you will probably want to know how to create more and more plots in parallel. Luckily we have a replay on YouTube of our cocktails with plotting experts. They had much to share about their various approaches. Some used servers and datacenter SSD, some bought used servers and SAS drives for temporary directories, some expand their consumer/gaming machines, and some focused on lots of smaller used machines. Many of them have compiled a spreadsheet of reference plotting hardware with plot speeds to help get you thinking about any hardware you might want to change or acquire and see how your plotting results measure up.
As you start parallel plotting you need to be careful to not over allocate memory when you are plotting. If you cause your operating system to swap, you are not going to be happy with your outcome. You don’t have to be as careful with thread count.
It is also a very common plotting strategy to plot on say your gaming machine and then move your plots to a Raspberry Pi 4 with a lot of USB ports. All you need is your same 24 word mnemonic on both machines. Alternatively you can just run a remote harvester on your Pi and have it connect to your gaming machine where you are running node and farmer and only have your private keys on one machine.
Learning more
Everyone trying to create plots should read through our repository FAQ. It really does answer 90% of the questions you might have about plotting (and farming.)
Once you have read the FAQ, you’ll find a supportive community in these channels on our public Keybase channels.
| Keybase Channel | Topic |
|---|---|
| #beginner | For those questions you are afraid to ask |
| #testnet | For all things testnet — an intermediate skill level |
| #plotting-hardware | The expert plotters are here. Hardware, software and plotting strategy |
Thanks
@pyl, @kiwihaitch, @psydafke, and @storage_jm all helped out on this post. The mistakes are mine. Should something need to be updated I will edit and post the errata down here.
Updates
As of Chia version 1.0.4, RAM min/med/max values have been updated.
Источник
Plotting with PowerShell on Windows
Windows PowerShell has developed into a great tool to manage and execute scripts on Windows machines. Its built into every install of Windows 10. In this post, we go into how to run plotters with PowerShell. There are a few advantages to running plotters in PowerShell:
- Plotting is not tied to the Chia GUI. If the Chia client experiences a crash or bug, your plotter will continue to work while the client recovers.
- Every PowerShell window is one plotter. If you plan to perform plotting in parallel (running multiple plotters on the same machine) then this is a perfect way to visibly see how many plotters are running and how far along they are.
- You can cancel a plotter queue after the current plot. Each plotter has its own queue length in PowerShell. Something that I discovered by accident, you can click the PowerShell window and press CTRL-C. Nothing will confirm the key command, but at the end of the current plot, the plotter will abort the queue.
- You can create a PowerShell script to run all your plotters. Possibly the best feature, you can run a script and all of your plotters are launched, with their own settings, in their own individual windows. Perfect.
There are a few house keeping rules first before using PowerShell so you don’t hit the pitfalls that I have. Perform these steps:
- Open Powershell by pressing the Windows key and typing “PowerShell”
- When the PS window opens, click the top left of the window to open a menu. In that menu, click “Properties.”
- In the new window, Uncheck the box that says “QuickEdit mode”. Normally this feature is great when you want to highlight text in the window and copy it out to another place. But, it has the bad side effect of pausing whatever is running in the window. This is bad if a plotter is plotting and you accidentally click the window while dragging it around. Disabling QuickEdit mode prevents this from happening. If you want to select text in the future, just right-click inside the PS window and select “Mark”.
- Next, click the “Layout” tab. Locate the “Height” field. This is how much scroll back you want. I like a ton of scroll back so I max this out at 9999. This is so that you can view the progress info of many plots to see times.
- Click the OK button once complete.
- If you ever plan executing your own PowerShell scripts, enter the command below in an Administrator PowerShell Window so that it is enabled:
Now that PowerShell is setup to prevent you from making my horrible mistake, we can proceed with how to get a plotter started. In your PS window, enter the following command:
In the command above, note the two bolded locations. The “ ” needs to be replaced with YOUR username on your machine. Next the “app-1.0.5” is going to be replaced with the version of chia that you are running. Currently the latest is 1.0.5. But it won’t be like this always.
After you Change Directories (cd) to that folder, enter the following command to begin a plotter:
There is a lot of information here so I’ll break it up:
- .\chia.exe plots create – This is pretty self explanatory. I want chia to create a plot.
- -k 32 – This is the size of plot you want to create. K=32 is the minimum size of a plot that is considered valid on the Chia Network. It is also the fastest to create. You can go smaller on the K value but those plots are not valid on the network.
- -b 3389 – This is the amount of ram you want to allocate to the plotter. It does not use all this ram at once, its just the limit. But, windows will preallocate this amount and call it “committed” which might push other applications to virtual memory on disk if committed is maxed. 3389 is the perfect amount if you are using 2 threads. I have found that 4 threads requires a minimum of 3408; 6 threads 3416; 8 threads 3424.
- -u 128 – This is the bucket size. Essentially its how many pieces you want the workload to be divided in. If you change this to 64, then you have to double the ram amount. From my experience, there is no change in plot speed messing with this number. So leave it at 128.
- -r 2 – This is the number of threads you want for the plotter. The plotter works in four phases. This number only affects phase 1 of a plot. Phase 2, 3, and 4 are all single threaded. In my testing, I saw a 30 minute speed improvement from using 2 threads to 4 threads. Then, I only saw a 5 minute improvement from 4 threads to 6 threads. There is diminishing returns definitely. Always try to use at least 2 though, because 1 thread is really slow.
- –t E:\temp – This is your temporary directory. Remember to pick a fast drive here.
- -d D:\plot – This is your final directory. This can be an external usb drive or another type of fat spinny hard drive.
- -n 1 – Finally this is your queue. How many plots do you want to make with these settings? Remember, you can set any number here and then stop it with CTRL-C.
- Since we are plotting on the same machine as the farmer, the keys needed to generate the plots are automatically brought in. So we don’t need to specify those here.
That’s pretty much it for the PowerShell command. Executing this command will start the plotter on its journey and it will print its progress in the window. Next is the scripting part. Below I have placed the script I use on my machine. Copy and paste this script into notepad and save the file as plotterscript.ps1, while selecting “All Files” for the file type:
Here is the script. Copy/paste the invoke-expression line for each plotter you wish to start :
Lets breakdown the script above. You will see semicolons in the script above. These separate each command of the new PowerShell window.
- invoke-expression ‘cmd /c start powershell -NoExit – This first part is what tells PowerShell to open another PowerShell window. The -NoExit flag is to not close the window when complete. Just in case you want to review the results.
- -Command
– The -Command here is the commands we want to pass to the new PowerShell window that opens up. - $host.ui.RawUI.WindowTitle = “t1p1” – This command allows you to rename the title of the PowerShell window. This is to keep track which window it is. Here, the name scheme I’m using is tempdrive 1, plotter 1. Since I have two plotter drives.
- start-sleep 0 – This next command is critical. When running plots in parallel you need to space them out a bit so that all of them are not trying to write to the final directory at the same time. If using an external drive, this causes a major headache. The value here is in seconds. For example, if you want an hour between plotters, it would be 0 for the first plotter, 3600 for the second plotter, 7200 for the third, etc.
- .\chia.exe plots create -k 32 -b 3389 -u 128 -r 2 -t E:\temp -d D:\plot -n 1 – This is the Chia plotter command that was explained above. Update this to your settings.
- Read-Host -Prompt “press enter to exit” – This was a hold-me-over from a previous version of the script incase for some reason the “-NoExit flag didn’t work.
There you have it, this should provide you with enough information in order to create and tweak your own scripts in order to maximize your plotters on your machine without having to worry about Chia Client issues.
With the file saved, all you need to execute it is to right click the file and select “Run with PowerShell”.
Источник