Loading…
2
A guy in a Discord said my training data was 'leaking' and I laughed at him for a week
Back in March I was fine tuning a small image model on my own scans and a guy named dev_null on a Discord kept telling me my validation set was too close to my training set. I ignored him since my accuracy numbers looked great, like 97 percent, and I figured he was just being a hater. Two weeks later I tried the model on real photos from a trip to Tucson and it fell apart completely, calling a cactus a lamp and a truck a fence. Turned out every single validation image had near duplicates in my training folder, so the model had basically memorized them. Anyone else get burned by a too clean val set before, and how do you check for overlap without eyeballing thousands of files?
0 comments
Log in to join the discussion
Log In0 Comments
No comments yet
Be the first to share your thoughts on this discussion.