Different Intervention Locations on Different Model Components

[comeandcode]

Hi, I am wondering if it is possible to intervene a model which contains both some layers of encoders and some layers of decoders. And I would like to assign different intervention locations for these two kinds of transformer blocks. I found that in the pyvene library, it just supports one intervention_locations argument. If it is possible to realize things like only prefix (7) for encoders and f7+l7 for decoders without modifying the original library? Thank you!

[PinetreePantry]

It is definitely possible to do this. See this section that creates interventions to different parts of the code. Beware that you also need to modify the intervention locations in the dataset file so that the intervention locations match the interventions.

Also note, decoder interventions are harder to implement correctly. You need to intervene on a variety length of input IDs, which complicates collating and padding. But theoretically there is nothing that prevents implementing encoder-decoder interventions. Pyvene/Pyreft is just a model intervening library that can do anything. Checkout my PR for an example.

[comeandcode]

Thank you for your reply! I will try it!