WARNING: The tool described in this repository is not supported in any way by Red Hat!!! |
---|
About#
This is a sample python client on top of the generated assisted-installer python library to ease working with assisted installer API
Available features:
create/delete cluster (autoinjecting pull secret file and ssh public key, static networking)
create/delete manifests
download discovery iso
wait for hosts
list cluster/hosts
update hostnames and host roles
update cluster attributes such as api vip and ingress vip
launch cluster install
wait for cluster
create day2 cluster for adding hosts
Deploying#
Offline token#
When using SAAS mode, an offline token is needed in order to interact with the api. This token can be retrieved at https://cloud.redhat.com/openshift/token
using rpm#
sudo dnf -y copr enable karmab/aicli ; sudo dnf -y install aicli
using pip#
Install the package with
pip3 install aicli
To upgrade it later on, use the following
pip3 install -U aicli assisted-service-client
Container mode#
alias aicli='podman run --security-opt label=disable --net host -it --rm -e AI_OFFLINETOKEN=$AI_OFFLINETOKEN -v $HOME/.aicli:/root/.aicli -v $PWD:/workdir quay.io/karmab/aicli'
Where AI_OFFLINETOKEN is an environment variable pointing to your saas offline token
With onprem mode, you can instead use -e AI_URL=$AI_URL
The container engine can also be docker.
How to use#
Configuration#
Setting target url#
by default, the tool targets the SAAS (which means providing an offline token)
Alternatively, You can indicate a target AI_URL, using the flag –url
The url needs to be provided in this format http://$AI_IP:8090
The target url can also be specified through environment variable AI_URL
the flag --staging
can be set to target the internal staging environment
Alternatively, set the env variable STAGING to true
In container mode, that would be -e STAGING=true
TLS settings#
You can use the flag ca, cert and key to provide ssl ca content, an ssl cert and key path to use for tls purpose
Alternatively, set the env variable AI_CA, AI_CERT and AI_KEY
Creating an onprem environment#
You can run the following command to deploy AI locally using podman:
aicli create onprem
The ip to use for the configuration is detected automatically using a socket connection to 8.8.8.8
If this is blocked, you can use pass -P ip=$your_ip
instead
You can also provide -P keep=true
to keep both configmap.yml and pod.yml so you can tweak them yourself
When no longer needed, the onprem deployment can be deleted using:
aicli delete onprem
Note that deploying AI this way is not the recommended approach for production/supported usage.
Basic usage#
the main objects you can interact with are cluster and hosts. For all of them, you can use create/delete/info/list/update subcommand
For most of the objects, you can provide either an id or a name.
Parameters#
For most of the commands, you can pass parameters either on the command line by repeating -P key=value
or by putting them in a parameter file, in yaml format.
If present, the file aicli_parameters.yml
is also parsed.
Typical workflow#
Interacting with the api should be straightforward but let’s walk through the typical steps you can use to deploy a cluster to completion
Note: consider using aicli create deployment
to deploy with a single step
Create cluster#
The basic way to create a cluster is to run:
aicli create cluster mycluster
When creating a cluster, two types of parameters can be provided, cluster keywords and extra keywords.
The cluster keywords are parameters exposed by AI api, while the extra keywords are goodies that provide shortcuts to set things such as sno, network_type,…
Those keywords can be listed with aicli list cluster-keywords
and aicli list extra-keywords
For instance, we can use the following command to create a cluster with a specific version and forcing the domain
aicli create cluster -P openshift_version=4.9 -P base_dns_domain=karmatron.local -P pull_secret=openshift_pull.json mycluster
This command will also use your default ssh public key so you don’t need to specify any (using ssh_public_key variable for instance)
To create a sno cluster
aicli create cluster -P sno=true mycluster
Within the extra parameters, the most common ones to use are listed below:
Parameter |
Meaning |
---|---|
api_ip |
Api ip |
ingress_ip |
Ingress ip |
domain |
Ingress ip |
minimal |
Whether to use minimal iso |
static_network_config |
the nmstate data to inject |
manifests |
a directory from where pick manifests to inject |
network_type |
Which sdn to use |
sno |
Whether to deploy a SNO |
sno_disk |
Which disk to use for SNO install |
hosts |
An array of hosts to automatically update data from |
pull_secret |
The path to your pull_secret (openshift_pull.json by default) |
ignore_validations |
Whether to ignore all validations |
Note: there are DNS requirements associated to the name of the cluster and the domain for an install to be available without /etc/hosts hacks
When a cluster gets created, an underlying infraenv named $cluster_infraenv also gets created under the hood.
In general, you shouldn’t have to care about this object, but notice it is actually where the iso information lives. The purpose of this object is to be able to boot nodes with a discovery iso without deciding initially on which cluster they belong (this is called late binding).
The nomenclature we use for the infraenv is consistent with what happens in AI UI, which means you can create a cluster and follow in the UI or use aicli to interact with a cluster created through the UI.
You can set the parameter infraenv to false to prevent an infraenv to get created for the cluster. You will then have to use the bind subcommand to associate hosts discovered through a given infraenv to some specific cluster. This same parameter can be provided when updating a cluster so that no action on infraenvs happen.
Custom ignition#
You can inject custom ignition in the discovery iso by either:
specifying
ignition_config_override
parameter with an ignition string
For instance,
ignition_config_override: "{\"ignition\":{\"version\":\"3.1.0\"},\"systemd\":{\"units\":[{\"name\":\"ca-patch.service\",\"enabled\":true,\"contents\":\"[Service]\\nType=oneshot\\nExecStart=/usr/local/bin/ca-patch.sh\\n\\n[Install]\\nWantedBy=multi-user.target\"}]},\"storage\":{\"files\":[{\"path\":\"/usr/local/bin/ca-patch.sh\",\"mode\":720,\"contents\":{\"source\":\"data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKc3VjY2Vzcz0wCnVudGlsIFsgJHN1Y2Nlc3MgLWd0IDEgXTsgZG8KICB0bXA9JChta3RlbXApCiAgY2F0IDw8RU9GPiR7dG1wfSB8fCB0cnVlCmRhdGE6CiAgcmVxdWVzdGhlYWRlci1jbGllbnQtY2EtZmlsZTogfAokKHdoaWxlIElGUz0gcmVhZCAtYSBsaW5lOyBkbyBlY2hvICIgICAgJGxpbmUiOyBkb25lIDwgPChjYXQgL2V0Yy9rdWJlcm5ldGVzL2Jvb3RzdHJhcC1zZWNyZXRzL2FnZ3JlZ2F0b3ItY2EuY3J0KSkKRU9GCiAgS1VCRUNPTkZJRz0vZXRjL2t1YmVybmV0ZXMvYm9vdHN0cmFwLXNlY3JldHMva3ViZWNvbmZpZyBrdWJlY3RsIC1uIGt1YmUtc3lzdGVtIHBhdGNoIGNvbmZpZ21hcCBleHRlbnNpb24tYXBpc2VydmVyLWF1dGhlbnRpY2F0aW9uIC0tcGF0Y2gtZmlsZSAke3RtcH0KICBpZiBbWyAkPyAtZXEgMCBdXTsgdGhlbgoJcm0gJHt0bXB9CglzdWNjZXNzPTIKICBmaQogIHJtICR7dG1wfQogIHNsZWVwIDYwCmRvbmUK\"}}]},\"kernelArguments\":{\"shouldExist\":[\"ipv6.disable=1\"]}}"
using the extra keyword
discovery_ignition_file
and providing a path where to store the ignition directly (that is, without specifying ignition_config_override)
Note that for such ignition to be applied to the nodes once they really get installed, a machineconfig is needed (which can be injected as custom manifest)
Custom kernel arguments#
You can specify kernel arguments that only apply to the discovery ISO by using the keyword kernel_arguments
.
The data can be provided either as a string (say a=xxx b=yyy
) in which case the arguments are considered to be appended or as a dict such as
kernel_arguments:
- operation: append
value: 'a=xxx'
- operation: append
value: 'b=yyy'
Custom install config#
You can tweak the install config of a cluster by providing a dictionary such as
aicli update cluster mycluster -P installconfig="{'CPUPartitioningMode': 'AllNodes'}"
Custom networking#
Network type#
Default CNI when creating a cluster is OVNKubernetes
the network_type extra keyword can be used to specify another one such as OpenshiftSDN, Calico, Contrail
Note that non standard CNIs typically have extra requirements of injecting custom manifests.
Tuning network of the nodes#
In order to use custom/static networking for your hosts, you need to provide nmstate information in the parameter file using the field static_network_config
You can also customize things such as cluster_networks, machine_networks and service_networks, for instance when trying to do a dual stack installation
You can find different samples here covering how to do:
static networking
bonding
dual stack
Registry url#
A non standard registry can be specified using the keyword registry_url
(or disconnected_url
) which is useful for disconnected install or testing okd.
Specify the corresponding ca content using the ca
variable or let it undefined to let aicli fetch this cert using openssl. This approach requires that there exists connectivity between aicli and your custom registry.
Adding extra manifests#
You can inject extra manifests (for instance if you are using a non standard CNI), for instance from the mydir directory, using the following commands
aicli create manifests --dir mydir mycluster
A flag allows you to have them stored in the openshift folder.
You can then use aicli list manifests mycluster
to confirm they were properly uploaded, or use aicli delete manifests
for deletion
Manifests from a given directory can be specified directly in your parameter file so that they get injected at cluster creation. For this, include the keyword manifests
and point it to the directory where your manifests are stored.
You can alternatively provide a list of entries below manifests such as
manifests:
- 10-xxx-ip.yaml: |
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: xxx-master
labels:
machineconfiguration.openshift.io/role: master
spec:
config:
ignition:
version: 2.2.0
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,IyEvYmluL3NoCgpOSUM9YnItZXgKSVA9JChhd2sgLUYgJ3JlbG9jYXRlaXA9JyAne3N1YigvIC4qJC8sICIiLCAkMik7IHByaW50ICQyfScgL3Byb2MvY21kbGluZSkKTkVUTUFTSz0kKGF3ayAtRiAncmVsb2NhdGVuZXRtYXNrPScgJ3tzdWIoLyAuKiQvLCAiIiwgJDIpOyBwcmludCAkMn0nIC9wcm9jL2NtZGxpbmUpCgpncmVwIC1xICRJUCAvZXRjL05ldHdvcmtNYW5hZ2VyL3N5c3RlbS1jb25uZWN0aW9ucy8qICYmIGV4aXQgMAoKY29ubmVjdGlvbj0kKG5tY2xpIC10IC1mIE5BTUUsREVWSUNFIGMgcyAtYSB8IGdyZXAgJE5JQyB8IGdyZXAgLXYgb3ZzLXBvcnQgfCBncmVwIC12IG92cy1pZiB8IGN1dCAtZDogLWYxKQpubWNsaSBjb25uZWN0aW9uIG1vZGlmeSAiJGNvbm5lY3Rpb24iICtpcHY0LmFkZHJlc3NlcyAkSVAvJE5FVE1BU0sgaXB2NC5tZXRob2QgYXV0bwppcCBhZGRyIGFkZCAkSVAvJE5FVE1BU0sgZGV2ICROSUMK
verification: {}
filesystem: root
mode: 448
path: /usr/local/bin/xxx.sh
systemd:
units:
- name: xxx.service
enabled: true
contents: |
[Unit]
After=network.target ovs-configuration.service
Before=network-online.target kubelet.service crio.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/xxx.sh
[Install]
WantedBy=multi-user.target
Ignoring validations#
For testing purposes, validations can be disabled when creating cluster by setting parameter ignore_validations
to true
Gather iso#
Once the cluster (and the corresponding infraenv) are created, we can get the discovery iso url using the following command
aicli info iso mycluster
or download it locally with
aicli download iso mycluster
Note that when AI api was in v1, a specific call create iso
was needed to trigger the creation of the iso, but it’s no longer needed (the command is maintained for retrocompatibility but does the same as info iso`
When using this call, the expiration time of the token associated to the iso is checked and if necessary, it gets refreshed (and as such so does the url)
Use a debugging user in the iso#
For troubleshooting purposes, typically when the host cant be accessed through ssh, you can add a user named aicli with password aicli in the discovery iso by running
aicli update cluster -P password=true mycluster
password boolean can be specified at cluster’s creation
Wait for hosts#
After booting some nodes with the iso, we normally wait for them to show up in the UI or in aicli list hosts
output.
Alternatively, we can use the following command to wait for 3 hosts to appear in mycluster
aicli wait hosts mycluster -n 3
Optionally Update hosts#
Once we have enough nodes, we need them show as known
in list hosts output in order to start the cluster deployment.
It might be necessary to update some specific information of the nodes, such as the requested hostname (localhost name is forbidden) or to assign a specific role to the nodes
Updating hostnames#
To change a specific host name, we can use the following
aicli update host $host -P requested_hostname=new_name
or simply
aicli update host $host -P name=new_name
If there are several matching hosts belonging to a same cluster, then the name is instead used as a prefix and the host names are sequentially assigned to name-0, name-1, …. That makes it easy to change all the localhost fqdns of your cluster with a single call
Updating roles#
To change the role of a given host to worker, you can run
aicli update $host -P role=worker
Updating extra args#
To specify extra args for a given host, you can run
aicli update $host -P extra_args="xxxx"
For instance, you can run the following to append kargs
aicli update host $host -P extra_args="--append-karg=rd.multipath=default --append-karg=root=/dev/disk/by-label/dm-mpath-root"
Specifying installation disk#
To specify installation disk for a given host, you can run
aicli update $host -P disk="xxxx"
For instance, To force installation disk to /dev/sdb, use
aicli update host $host -P disk=sdb"
Skipping formatting specific disks#
To avoid formatting of given disks, you can run
aicli update $host -P skip_disks=[disk1,disk2]
For instance, To skip formatting of disks /dev/sdb and /dev/sdc, use
aicli update host $host -P skip_disk=[sdb,sdc]"
Note: the parameter skip_formatting_disks
can be used instead but you will need to provide a dict with disk_id and skip_formatting
Specifying node labels#
To declare specific labels for a host, you can run
aicli update $host -P labels=[label1,label2,key3=value3]
For instance, To specify the label productionready
for a host, use
aicli update host $host -P labels=[productionready]"
Note: the parameter node_labels
can be used instead
Applying custom ignition on first boot of a host#
We can use the extra keyword ignition_file
and provide a path where to store the ignition directly and call
aicli update host $host -P ignition_file=your_ignition_path"
Updating from a parameter file#
You can specify in your parameter file a hosts array so that the information for updating hosts is gathered from there.
For instance, if you have the following information in your parameter file
hosts:
- name: xxx.fantastic.com
role: master
- name: yyy.fantastic.com
role: worker
extra_args: "ip=dhcp6"
Running aicli update hosts --paramfile my_params.yml
will change the roles of the hosts with the corresponding name, if found, and add the specified extra_args for the second host.
You can either use the name to match a host, or specify id (to match system uuid) or mac (to match any mac address from the host inventory) so that the name of the host can be forced.
When using the create deployment workflow, update hosts is run as one of the steps.
Updating cluster#
At this step, you might need to update cluster data so that the cluster is ready to install.
For instance, you might want to specify api vip and ingress vip now that hosts cidrs have been discovered.
For this, you can run
aicli update cluster -P api_ip=$api_ip -P ingress_ip=$ingress_ip mycluster
Launch cluster deployment#
Once your hosts all show as known, the cluster status should appear as ready in aicli info cluster mycluster
At this point, you can trigger the deployment using the following command
aicli start mycluster
Monitor deployment#
Wait for cluster#
When the cluster is installing, you can wait for it to complete using the following command
aicli wait mycluster
Monitor events#
You can also see all events associated to your cluster using
aicli get events cluster
Gather assets#
Once installation has started, you can gather relevant assets for your cluster such as
kubeconfig
kubeadmin-password
installconfig
For instance, to gather the kubeconfig, you can use the following to get it downloaded to your current directory as kubeconfig.mycluster
aicli download kubeconfig mycluster
Add extra workers#
For this purpose, we assume we already have the cluster installed.
If we create a new cluster with the same original name and the ‘-day2’ extension, the api code will create a dedicated cluster for adding host purposes.
aicli create cluster mycluster-day2
Alternatively, we can convert the original cluster into one to use for adding hosts by using the following call but note this is not the recommended approach
aicli update cluster mycluster -P day2=true
Note that when creating the day2 cluster, a DNS check on api_vip_dnsname is done. If it doesn’t succeed and the base cluster is HA, then api vip is used instead of fqdn to guarantee functionality
You can also update manually this data using the following command
aicli update cluster mycluster-day2 -P api_vip_dnsname=$api_ip
In both cases, once we have the day2 cluster in the proper state, the same workflow is to be used:
gather the iso associated to this cluster/infraenv with
aicli info iso mycluster-day2
.boot nodes with this iso.
wait for them to show in
aicli list hosts
output as knownlaunch
aicli start cluster mycluster-day2
Individual hosts installation can also be triggered by calling:
aicli start hosts $host1 $host2 ...
Deployment workflow#
Instead of deploying the cluster step by step, you can put all the relevant information in your parameter file and then have all the steps run for you
You can use a command such as the following one
aicli create deployment --paramfile my_params.yml myclu
The parameter file could be similar to the following one
base_dns_domain: karmalabs.com
api_vip: 192.168.122.253
ingress_vip: 192.168.122.252
download_iso_path: /var/www/html
download_iso_cmd: "chown apache.apache /var/www/html/ci-ai.iso"
iso_url: http://192.168.122.1/ci-ai.iso
bmc_user: admin
bmc_password: password
hosts:
- name: ci-ai-ctlplane-0
bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111181
- name: ci-ai-ctlplane-1
bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111182
- name: ci-ai-ctlplane-2
bmc_url: http://192.168.122.1:8000/redfish/v1/Systems/11111111-1111-1111-1111-111111111183
Note that in this case, we are providing bmc information for our hosts so that they get booted with the discovery iso automatically.
We also have the iso downloaded automatically to a path corresponding to a web server
If you omit this kind of information, you can still have the deployment done semi automatically by just waiting for the iso url to be displayed and plug it manually to your target nodes.
Ansible Integration#
aicli modules are available at karmab/ansible-aicli-modules and provide the following primitives:
ai_cluster
ai_cluster_info
ai_infraenv
ai_infraenv_info
ai_host_info