So, I have a shot which has two subjects (dressed as soldiers wearing green) standing in a field surrounded by greenery in tall grass. Currently trying to composite an element between them and the tall grass. Is this possible to do in a clean way? I have tried luma mattes, threshold tricks, extraction, and a few other things. Is there a clean way to separate these guys and the tall grass in front of them? It seems as though something like the refine edge tool might do the trick if it were applicable but I'm not quite sure if it can be implemented here. Looking for any help, hopefully there's a trick/technique I'm missing. Harder things have been Roto'd. I just need to figure out how.
Without seeing your footage it's hard to say precisely as every challenging composite needs a unique approach, however, a few things come to mind and forgive me if you've already attempted these.
Have you tried duplicating your footage layer and pushing the contrast or saturation on the duplicated layer to the point where you can achieve a reasonable luma or chroma key, then applying refine matte or similar to smooth the inevitably hard edges and use the resulting composite as an alpha matte on the original footage. If this doesn't work you might get better results by separating out the RGB channels and using the same technique on the single colour channel that best separates the grass and your subjects.
Alternatively, can you cheat it? Is there a part of the image that offers better luma or chroma separation from the grass, such as against a bright sky that you can key? With a feathered mask, some colour correction and a little distortion you could use a duplicate of this area over the soldiers legs to hide any issues with your composite.
Also, where tools such as keylight fail because of similar colour or luminosity values, going old school and using multiple instances of linear colour key with extremely low threshold and softness settings so you are removing very narrow bands of colour with each instance can give more accurate results. As with the first technique this will leave you with blocky edges that will need to be fixed with some kind of matte/edge refine
These are all techniques that have worked for me in the past when I have been struggling to separate subjects and roto is not really an option.
Using multiple layers that use chroma and luma keys individually to isolate different areas and then using blend modes such as 'darken' or 'soft light' etc to overlay each one is a technique I've had success with specifically on shots with long grass.
Hope there's something there that might offer some insight.