Chia last attempted proof plots passed filter

None of your plots have passed the plot filter yet. You are currently farming 1.089 TiB #3035

Comments

lpenrod commented Apr 28, 2021

I have 2 chia rigs set up.
Both Windows 10
*chia1 is the master full node — 192.168.1.30
*chia2 is a second full node — 192.168.1.29

I moved the CA from chia1 to chia2 and did the chia init -c [directory] and that was successful.

chia1 I originally opened TCP port 8447, but during trouble shooting, it is now open TPC/UDP ANY ANY

chia2 I’ve disabled UPNP and added under Harvester, farmer_peer, host:, the IP of the chia 1 machine.

Both rigs sync their block-chain. Both rigs are downloading plots (cli). Both rigs say they are Farming. chia1 is regularly receiving «Last Attempted Proof» and «Latest Block Challenges». chia2 under farming says «None of your plots have passed the plot filter yet. But you are currently farming 1.089 TiB»

I’ve ran TCPview on both machines. I see a connection from chia2 to chia1 on port 8447 and packets sent and recieved.

I’ve changed the logging to both machines to info.

Looking at the logs chia2 I see:
2021-04-28T14:38:38.016 harvester harvester_server : INFO farming_info to peer 192.168.1.30
I see the above over and over.

Looking at the logs of chia1 I see:
2021-04-28T15:06:23.461 farmer farmer_server : INFO -> new_signage_point_harvester to peer 192.168.1.29 [value]
2021-04-28T15:06:23.461 farmer farmer_server : INFO mainnet->db folder and restarted.

All along the way I have stopped and started chia on both servers. They do their handshake and then communicate on port 8447.

I’m lost at this point.

The text was updated successfully, but these errors were encountered:

Источник

Nothing tells you when you’re timing out on 30 second harvester proof checks #2651

Comments

coding-horror commented Apr 25, 2021 •

The problem

I thought I was farming, but I wasn’t — because something about my network caused the proof check to take more than the hard-coded 30 second limit.

I had an average time to win of 8 or 9 hours for more than 120 hours without a single win. This seemed statistically implausible, so I researched the logs, and cleared any errors or warnings in the logs (well done, all the warnings and errors in debug.log were indeed things I should fix!). Still no wins for a long time.

How to reproduce

Have a bunch of plots on slow storage media; when the proof check happens, verifying the proofs takes longer than the hard-coded 30 seconds allowed. You will never win a single Chia, but there’s absolutely nothing in the GUI to inform you that this is happening. You can view the logs, but in the logs it is not even presented as a warning (!), but as an INFO message:

Of the above, the proofs that take longer than 30 seconds are not eligible to win, but this is not logged as an ERROR or WARNING or surfaced in the UI in any way.

Expected behavior

The GUI will tell you «hey, your proof checks are too slow, there’s absolutely no chance for you to win, even if you are farming infinity plots»

Screenshots

Desktop

• OS: Windows 10
• OS Version/Flavor: latest 19042.928
• CPU: i7-1165G7

Additional context

I followed up on the #support channel in Keybase, where I got the important advice to enable INFO level logging and check for the 30 second proof limit.. and I wrote up a detailed account on the forum; if you need excruciating levels of detail, please check there 🙇‍♂️

Recommended solution

• Allow the 30 second proof check threshold to be configurable to account for slower storage (or relax it a bit)
• change the log level of beyond 30 second on harvester proof checks to WARN or ERROR
• in the logs, provide details on which files are being checked for proofs, so you can identify which media or NAS is potentially the problem one slowing everything down

The text was updated successfully, but these errors were encountered:

maomingyang1314 commented Apr 25, 2021

Moreover, I would like to know why your laptop can mount nearly 400TB disks, is your disk on nas?

maomingyang1314 commented Apr 25, 2021

Your NAS caused your task to time out. It is recommended to abandon this method.

AZ663 commented Apr 25, 2021

mariano54 commented Apr 25, 2021

We can’t change the consensus algorithm anymore, but we can change the log level, and show the files.

Please note that looking up qualities for plots passing the filter requires about 7 random reads in a plot, whereas actually looking up a proof requires 64 reads. It might not be feasible on a slow NAS, since these are sequential reads. Furthermore, you need to take into account network latency to propagate your proof and block to the network, so you should be under 5 seconds to reduce risk of losing rewards.

Actually, the proof of space library does them sequentially, but they could be done in parallel, since it’s a tree, so you could do 1 read, then 2, then 4, .. etc, for a total of around 7 sequential phases (one for each table in the plot). We haven’t got around to doing this yet.

mariano54 commented Apr 25, 2021

Also another thing to point out is that the responses are returned to the full node as they come out from the drives, so the high time is probably only affecting the slow drive or the slow NAS.

coding-horror commented Apr 25, 2021

Yeah, this only started happening as I added multiple NAS devices to the network. With 1 or 2 NASes, it was all fine. Once you get to 5.. not so much, especially if the algorithm picks plots on 6 different devices. At the very least

• definitely change the log level to ERROR (or WARN)
• surface this in the UI for sure, otherwise you’re silently «farming».. absolutely nothing. A silent killer.
• tell us WHICH files were picked for proofs so we can look for «problem» devices

immortalx commented Apr 26, 2021

Is there any benefit of using larger plot files in this case? Less files per directory in his case.

keliew commented Apr 26, 2021

I think the problem is inherent in how Windows manages mounted network drives.

If you’d to switch to a Linux-based farmer, it should work better. I know it’s not an option for most people, but it’d be a good test to see if that’s the cause.

kimklai commented Apr 26, 2021

I think lots of people need to know their whole system just too slow to provide the a valid answer in time. So please mark the logs as WARNING as they takes longer a certain threshold.

desek commented Apr 26, 2021

This was quite a rough finding since I’ve been happily farming on my RPi on wifi to a remote storage with proof checking usually between 60-90 seconds.

tartley commented Apr 26, 2021

Could some drives be powering down when idle?

bertybassett commented Apr 27, 2021 •

yeah my 1 Synology NAS is doing the same if I hit it hard with other services such as docker, Plex, etc

My Synology can sleep the HDs and out of the box I believe it set to that as default, no idea about yours.

Synology HDD hibernation
Synology NAS, expansion units, and USB HDDs enter different energy-saving modes after being idle. . Phase 1: HDDs power down after a period of inactivity (idle time). You may go to DSM > Control Panel > Hardware & Power > HDD Hibernation to configure the length of inactivity for the HDDs to enter hibernation.

cccat6 commented Apr 27, 2021 •

We can’t change the consensus algorithm anymore, but we can change the log level, and show the files.

Please note that looking up qualities for plots passing the filter requires about 7 random reads in a plot, whereas actually looking up a proof requires 64 reads. It might not be feasible on a slow NAS, since these are sequential reads. Furthermore, you need to take into account network latency to propagate your proof and block to the network, so you should be under 5 seconds to reduce risk of losing rewards.

Actually, the proof of space library does them sequentially, but they could be done in parallel, since it’s a tree, so you could do 1 read, then 2, then 4, .. etc, for a total of around 7 sequential phases (one for each table in the plot). We haven’t got around to doing this yet.

@mariano54
My disk array is not in the local but distributes in different places. I gather them as a huge disk and connect them with NFS. The latency of this solution for each random seek is between 200-1000ms. The average and the mean is about 400ms.
I wondering whether this is able to farm. 400ms*(7+64)seeks will be 28.4s. Just a little bit below the 30s limitation.
However, this solution support parallel read. If it is seeking 64 different places in the plot at the same time, it will be able to respond to all of them within a single seek time, which is about 400ms.
Is parallel seeking in the todo list and is that work as what I am thinking? When it will be updated?
Thanks a lot!

Источник

[help] full node stops proofing its plots and not showing itself in harvester network #3810

Comments

drewmullen commented May 5, 2021 •

UPDATED ISSUE DESCRIPTION

The original issue setup has been changed per discussion but the same issue still persists. The latest description of the issue should be found HERE. Basically i stopped using a NAS but the problem described below still happens; harvester process goes defunct with no errors found in the logs

ORIGINAL ISSUE DESCRIPTION

Describe the issue

• 1 full node — windows, gui, plotting and farming a NAS
• 2 plotters/harvesters (fedora)
• 1 only plotter (ubuntu, copying to nas)

all are on version 1.1.4 . after a couple hours the windows node:

• stops showing accurate count of plots on the nas
• stops showing plots its responsible for farming in Last Attempted Proof
• stops showing itself ( 127.0.0.1 ) in «Your Harvester Network»

GUI does show the harvester node IPs in Harvester network & it does show their plot counts in Last attempted proofs

I dont see any obvious errors with Get-Content

I’ve let it run for hours and it stays the same. a restart of the app resolves all 3 of those bullets

should i be concerned? is there a way i can validate that the full node IS actually farming those plots & attempting proofs with them?

The text was updated successfully, but these errors were encountered:

dnavamosler commented May 5, 2021

same problem here

dorkmo commented May 5, 2021

NAS has been reported to cause farming read delays.

drewmullen commented May 5, 2021 •

NAS has been reported to cause farming read delays.

Are you suggesting that:

• this is an issue i should be concerned with?
• using a NAS will cause this issue?

—- additional info below

pics of what im experiencing

stops working (there should now be 272 in the full node plot count):

drewmullen commented May 5, 2021

i moved most of my plots off the full node NAS and plugged directly to a harvester. i still have some plots on the nas though so we can troubleshoot. this feels like a bug to me though. either a GUI bug (not reporting accurately) or a really nasty bug causing me to lose a ton of farm time on plots

dorkmo commented May 5, 2021

yeah if those 270 arnt passing the filter you might have a delay issue. I dont have a nas so not sure, but you might search NAS in the github repository issues/discussipns.

drewmullen commented May 5, 2021

thanks for the advice @dorkmo . i think i can close this for now. maybe someone will solve the NAS issues at some point but im fine with plugging them into my harvester

drewmullen commented May 5, 2021 •

This definitely smells like a bug. after moving from farming via nas to plugging directly into a harvester, i get the same issue, harvesters with larger plot counts stop reporting in my gui

last reported item from debug.log (1 hour old)

@dorkmo any idea how we can get the attention of one of the devs / support engineers to help troubleshoot further? should i change [help] to [bug] ?

dorkmo commented May 5, 2021

might read over this thread #3188 too

might have lost connection between node and harvester? are you able to plug your NAS into your node computer via usb?

does the harvester have a log? unfortuantely i’ve got a real basic setup so i dont know much about advanced stuff. not sure how to get help, i’d just try to gather as much info as possible and post here.

drewmullen commented May 5, 2021 •

after you pointed out the issue with NAS’ i moved all those external hdds and plugged them directly into a harvester (fedora). Here is the adjusted setup:

• 1 full node: windows, managing about 30 plots that are on a nas
• 1 harvester: fedora, managing about 325 plots
• 2 other harvesters (not important)

after i switched from having

300 plots managed on the NAS to plugged directly into the fedora box, i figured that would solve the issue, closed the ticket. HOWEVER, i actually saw the exact same behavior. the harvester with 300+ plots was dropped from my local harvester network, and i was no longer getting proof attempts from those plots. thats what this post describes:

in the full node gui, the harvester with 320 plots is:

• no longer represented in Last Attempted Proof (only shows 10 / 30 from diff harvesters)
• no longer represented in «Your Harvester Network»

Источник