Please read more detailed FAQs about Rancher Server and Rancher Agent/Hosts.
This section assumes you were able to successfully start Rancher server and add hosts.
Docker containers are immutable (not changeable) after creation. The only things you can edit are things that we store that aren’t really part of the Docker container. This includes restarting, it’s still the same container if you stop and start it. You will need to remove and recreate a container to change anything else.
You can Clone, which will pre-fill the Add Container screen with all the settings from an existing container. If you forget one thing, you can clone the container, change it, and then delete the old container.
In Docker, linked containers (using --link in docker run) shows up in the /etc/hosts of the container it’s linked to. In Rancher, we don’t edit the /etc/hosts. Instead we run an internal DNS server that makes links work across hosts. The DNS server would respond with the correct IP.
Since the agent is potentially open to the public internet, requests to the agent for a shell (or logs, etc) of a container aren’t automatically trusted. The request from Rancher Server includes a JWT (JSON Web Token) and that JWT is signed by the server and can be verified by the agent to have actually come from the server. Part of that includes an expiration time, which is 5 minutes from when it is issued. This prevents a token from being used for long periods of time if it were to be intercepted, which is particularly important if not using SSL.
If you run docker logs -f rancher-agent and the logs show messages about an expired token, then please check that the date/time of the Rancher Server host and Rancher Agent host are in sync.
In the service details, we provide service logs in a tab called Log. In the Log tab, it lists out all events related to the service including a timestamp and description of the event that occurs in the API. These logs are kept for 24 hours before being deleted.
If containers on different hosts cannot ping each other, there are some common scenarios that could be the issue.
In the Stacks -> Infrastructure Stacks, check the status of the healthcheck stack. If the stack is active, then cross host communication is working.
To manually test, you can exec into any container and ping the internal IP (i.e. 10.42.x.x) of another container. The containers from infrastructure stacks may be hidden on the hosts page. To view them, select the “Show System” checkbox in the upper right corner.
Every so often, the IP of the host will accidentally pick up the docker bridge IP instead of the actual IP. These are typically 172.17.42.1 or starting with 172.17.x.x. If this is the case, you need to re-register your host with the correct IP by explicitly setting the CATTLE_AGENT_IP environment variable in the docker run command.
$ sudo docker run -d -e CATTLE_AGENT_IP=<HOST_IP> --privileged \
-v /var/run/docker.sock:/var/run/docker.sock \
rancher/agent:v0.8.2 http://SERVER_IP:8080/v1/scripts/xxxx
If you have UFW enabled, you can either disable UFW OR change /etc/default/ufw to:
DEFAULT_FORWARD_POLICY="ACCEPT"
10.42.0.0/16) used by Rancher is already used in my network and prohibiting the managed network. How do I change the subnet?The default subnet used for Rancher’s overlay networking is 10.42.0.0/16. If your network is already using this subnet, you will need to change the default subnet used in Rancher’s networking. You will need to ensure the networking infrastructure service that you want to use has the correct subnet in the default_network in the rancher-compose.yml file.
To change Rancher’s IPsec or VXLAN network driver, you will need to have an environment template with an updated infrastructure service. When creating a new environment template or editing an existing one, you edit the configuration of the networking infrastructure service by clicking on Edit Config. In the edit screen, you can enter a different subnet in the Configuration Options -> Subnet and click Configure. Any new environment using the updated environment template would be using the new subnet. Editing an existing environment template will not update the infrastructure services in existing environments.
Here’s an example of the updated network driver’s rancher-compose.yml to change the subnet to 10.32.0.0/16.
ipsec:
network_driver:
name: Rancher IPsec
default_network:
name: ipsec
host_ports: true
subnets:
# After the configuration option is updated, the default subnet address is updated
- network_address: 10.32.0.0/16
dns:
- 169.254.169.250
dns_search:
- rancher.internal
cni_config:
'10-rancher.conf':
name: rancher-cni-network
type: rancher-bridge
bridge: docker0
# After the configuration option is updated, the default subnet address is updated
bridgeSubnet: 10.32.0.0/16
logToFile: /var/log/rancher-cni.log
isDebugLevel: false
isDefaultGateway: true
hostNat: true
hairpinMode: true
mtu: 1500
linkMTUOverhead: 98
ipam:
type: rancher-cni-ipam
logToFile: /var/log/rancher-cni.log
isDebugLevel: false
routes:
- dst: 169.254.169.250/32
Note: The previous method of updating the subnet through the API will no longer be applicable as Rancher has moved to infrastructure services.
If you want to see the configuration of the Rancher DNS setup, go to the Stacks -> Infrastructure. Find the network-services stack and select the metadata service. In the metadata service, exec into any of the containers named network-services-metadata-dns-X. You can use the UI and select Execute Shell on the container.
$ cat /etc/rancher-dns/answers.json
If you run a container on the host (i.e. docker run -it ubuntu) and the container cannot talk to the internet or anything outside the host, then you might have hit a networking issue.
CentOS will by default set /proc/sys/net/ipv4/ip_forward to 0, which will essentially bork all networking for Docker. Docker sets this value to 1 but if you run service restart networking on CentOS it sets it back to 0.
Initializing>?Load balancers automatically have health checks enabled on them. If the load balancer is stuck in initializing state, then most likely the cross host communication between the hosts is not working.
If you want to see the configuration of the load balancer, you will need to exec into the specific load balancer container and look for the configuration file. You can use the UI and select Execute Shell on the container.
$ cat /etc/haproxy/haproxy.cfg
This file will provide all the configuration details of the load balancer.
The logs of HAProxy can be found inside the load balancer container. docker logs of the load balancer container will only provide details of the service related to load balancer, but not the actual HAProxy logging.
$ cat /var/log/haproxy
As of Rancher v1.6.11
In order to enable the logging from HAProxy, you’d have to update the custom configuration of the load balancer with the following defaults:
defaults
log 127.0.0.1:8514 local0 debug
In an HA set, if rancher-compose-executor and go-machine-service are continuously restarting, if you are behind a proxy, please ensure that proxy protocol is being used.
If something goes wrong with your authentication (like your GitHub authentication getting corrupted), then you may be locked out of Rancher. To re-gain access to Rancher, you’ll need to turn off Access Control in the database. In order to do so, you’ll need access to the machine that is running Rancher Server.
$ docker exec -it <rancher_server_container_ID> mysql
Note: The
<rancher_server_container_ID>will be the container that has the Rancher database. If you upgraded and created a Rancher data container, you’ll need to use the ID of the Rancher data container instead of the Rancher server container.
Access the cattle database.
mysql> use cattle;
Review the setting table.
mysql> select * from setting;
Update the api.security.enabled to false and clear the api.auth.provider.configured value . This change will turn off access control and anyone can access Rancher server with the UI/API.
mysql> update setting set value="false" where name="api.security.enabled";
mysql> update setting set value="" where name="api.auth.provider.configured";
Confirm the changes have been made in the setting table.
mysql> select * from setting;
It may take ~1 minute before the authentication will be turned off in the UI, but you will be able to refresh the webpage and access Rancher with access control turned off.