Provide tooling for importing existing RouterOS configuration easily

[ricardbejarano]

Is your feature request related to a problem? Please describe.
Onboarding pre-configured devices into Terraform is tedious, given most resources are imported using their *N IDs.

Describe the solution you'd like
Have a script that connects to a RouterOS device and generates the Terraform code needed to import all its configuration into resources, along with all non-default resource settings.
It doesn't need to structure the resources into state perfectly, that can be refactored later, but getting the resources into state at the very least would be fantastic for devices with 100+ settings in deployments with multiple devices.

I am willing to develop this myself, I'm opening the issue to gather feedback and track progress.

[vaerh]

I think that would be a very relevant tool. Since provider users periodically need to import complex configurations.
For my part, I can get my act together and complete the move to a flexible resource import function (where we can import by specifying any unique attribute). Partially it is already done, but not for all resources.
Also, in my opinion, you will need to interact with the provider or source code to be able to extract at least “Required” attributes and resource names. So that correct initial configurations can be obtained.

[ricardbejarano]

Yeah, that's the idea, get into the provider (code if possible) and extract the schemas and import "instructions".

Then generate HCL with import and resource blocks for every setting that would show up in an /export run.

Doesn't need to be perfect. It's okay if it only supports 90% of resources. It's okay if the code/state structure it proposes isn't perfect, the user can later use moved blocks to refactor into their desired structure.

[vaerh]

@ricardbejarano
Please test this

[ricardbejarano]

Wow! That was fast! I'll plug it in tomorrow morning. Thanks!

[vaerh]

Of course, there's no rush.
There are a few things that need SSH, as I have not been able to get the export output easily in any other way.
And you need to test on a large configuration, as the processing of some resources will have to be customized.

I hope for your help. Do you speak Go? :)

[ricardbejarano]

Okay so I've tested it and it works. Thanks for putting this together.

Some notes:

• Where possible, I'd offload as much logic about SSH's configuration to the environment's configuration (.ssh/config, ssh-agent, etc.).

  • That will make this more compatible with other setups while keeping it simpler -in the long run- than support every single ssh setting.
  • I emphasize on this because some may be using private keys instead of passwords, others might be hopping through a bastion, and there's always someone who will try to do this via a UNIX sockets.

• Instead of writing a terraform import script, use import blocks.

  • First, this means only one instantiation of the provider and one state update, whereas a terraform import script would have as many instantiations and updates as there are lines in the script.
  • Second, we can put the import blocks right next to their resource counterpart, making it easier for the user to read the generated output.
  • Third, now that we're putting import and resource together, I'd also add the original /export line for the resource in a Terraform comment.
  • And finally, this leaves us with just one file to write (the configuration file), meaning we can also pluck that logic off this code and print to stdout, simplifying the tool and enabling the user to redirect (> routeros.tf) or pipe (| whatever) it freely.

• The two most crucial pieces of this tool are:

  • The logic matching /export lines to provider resources. This must be bulletproof and scalable.
    • Ideally, all Terraform resources' names should match the RouterOS feature replacing slashes with underscores, e.g.: /ip/dhcp-server/lease → routeros_ip_dhcp_server_lease. So far I believe this is the case, but haven't checked thoroughly.
    • Otherwise, if not possible or to retain backwards compatibility, maybe extend all resource definitions with a "ExportParser" function, which we can then use from this tool, but it needs to be next to the resource definition, so that people are expected to implement it when onboarding a new resource.
  • The logic that finds IDs. This also must be bulletproof and I think is the hardest problem of the two.
    • Perhaps a couple default reusable functions (one like the one you implemented, another one for resources whose import id is .), then allow that function to be overriden on a per-resource basis if they have more complex/inconsistent id resolution instructions.

I think that's all I have for now. This looks good, and will definitely save a lot of time to a lot of people.

PS: i do speak Go, and can (and am willing to) help once we're aligned on the direction.

[ricardbejarano]

I've also been thinking about multi-device scenarios, whether we would want to support a list of devices instead of just one, but I think it's best if we leave that for-looping logic to the user's shell.

So if the tool ends up looking something like this:

terraform-provider-routeros-importer my-router.local > routeros.my-router.tf

Then we should just let users do something like:

for device in a b c; do
  terraform-provider-routeros-importer "${device}.local" > "routeros.${device}.tf"
done

And avoid overcomplicating the tool with multi-provider support.

We could, however, support a --provider-alias flag to set the provider meta-argument of the resources generated by the tool, so that users don't have to override that manually on each resource after code has been generated.

[vaerh]

I'll also answer on points:
There is already well written code for ssh that covers all possible cases. I suggest not to waste time on it now and add it at the end, of course if you do not use certificates to access MT. We can just take any suitable module.

I am absolutely not against changing the logic of resource output, be it a file or stdout. At the moment this is a draft version convenient for debugging.
The import sections are very interesting, thanks I missed adding them to TF. But you have to be prepared for the fact that when importing sections you may have a problem (wrong attributes, unfilled values or something else). In that case, it would be better to go through this scenario step by step and it might be worthwhile to provide both ways of importing, given that it's not difficult at all.

On finding a match between MT paths and TF resources, I suggest tackling the problem as it comes up, as the available method should cover most of the resources present. A few entries may need to be added, but this needs to be checked.

Searching for resource identifiers doesn't seem difficult to me either, since the TF schema easily contains required attributes that can be used to uniquely identify any resource. But even in this case there may be exceptions (you can see it in the code of the Provider in the resource creation functions).
Identifiers . are used for importing system resources (in provider's terminology). This point is optional and can be replaced by any identifier. Such resources are imported simply by their path.

I want to warn you about a few limitations that I found out while making this concept:

• To limit the list of attributes that will be populated in the HCL sections it is best to use the output of the /export command. This way only added or changed attributes will be displayed. Otherwise MT returns more data than necessary.
• To get /export output we can use SSH or a combination of exporting to a file and downloading it to ourselves (this is implemented via API/REST, though not as easy as SSH).
• When working via SSH (querying the CLI) you should be very careful with the returned data, e.g. the format of returned time attributes will not match the REST/API data.
• When processing /export output we will have some resources that we have not configured (notes) and the HCL configuration will still need to be redacted.
• It is not possible to use dynamic retrieval of the provider schema via terraform providers shcema because in this case there will be no default attribute values and no way to retrieve resource paths (__path__).