Quantcast
Channel: Shinguz's blog
Viewing all 318 articles
Browse latest View live

First Docker steps with MySQL and MariaDB

$
0
0

The Docker version of the distributions are often quite old. On Ubuntu 16.04 for example:

shell> docker --version 
Docker version 1.13.1, build 092cba3

But the current docker version is 17.09.0-ce (2017-09-26). It seems like they have switched from the old version schema x.y.z to the new year.month.version version schema in February/March 2017.

Install Docker CE Repository

Add the Docker's official PGP key:

shell> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

Add the Docker repository:

shell> echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"> /etc/apt/sources.list.d/docker.list
shell> apt-get update

Install or upgrade Docker:

shell> apt-get install docker-ce
shell> docker --version
Docker version 17.09.0-ce, build afdb6d4

To test your Docker installation run:

shell> docker run --rm hello-world

Add Docker containers for MariaDB, MySQL and MySQL Enterprise Edition

First we want to see what Docker containers are available:

shell> docker search mysql --no-trunc --filter=stars=100
NAME               DESCRIPTION                                                                                         STARS OFFICIAL AUTOMATED
mysql              MySQL is a widely used, open-source relational database management system (RDBMS).                  5273  [OK]
mariadb            MariaDB is a community-developed fork of MySQL intended to remain free under the GNU GPL.           1634  [OK]
mysql/mysql-server Optimized MySQL Server Docker images. Created, maintained and supported by the MySQL team at Oracle 368            [OK]
percona            Percona Server is a fork of the MySQL relational database management system created by Percona.     303   [OK]
...

OK. It seems like MySQL Server Enterprise Edition is missing. So we have to create an account on Docker Store and get the MySQL Server Enterprise Edition Image from there:

shell> docker login --username=fromdual
Password:
Login Succeeded

Unfortunately one can still not see MySQL Server Enterprise Edition.

But we can try anyway:

shell> docker pull store/oracle/mysql-enterprise-server:5.7
shell> docker logout
shell> docker pull mysql
shell> docker pull mariadb
shell> docker pull mysql/mysql-server

To see what is going on on your local Docker registry you can type:

shell> docker images
REPOSITORY                           TAG    IMAGE ID     CREATED       SIZE
mariadb                              latest abcee1d29aac 8 days ago    396MB
mysql                                latest 5709795eeffa 2 weeks ago   408MB
mysql/mysql-server                   latest a3ee341faefb 5 weeks ago   246MB
store/oracle/mysql-enterprise-server 5.7    41bf2fa0b4a1 4 months ago  244MB
hello-world                          latest 48b5124b2768 10 months ago 1.84kB

I personally do not like that all those images which are tagged with latest because I want a clear control over what version is used. MariaDB and MySQL community server have implemented this quite nicely but not MySQL Enterprise Edition:

shell> docker pull mariadb:10.0
shell> docker pull mariadb:10.0.23
shell> docker pull mysql:8.0
shell> docker pull mysql:8.0.3

docker images | sort
REPOSITORY                           TAG     IMAGE ID     CREATED       SIZE
hello-world                          latest  48b5124b2768 10 months ago 1.84kB
mariadb                              10.0.23 93631b528e67 21 months ago 305MB
mariadb                              10.0    eecd58425049 8 days ago    337MB
mariadb                              latest  abcee1d29aac 8 days ago    396MB
mysql                                8.0.3   e691422324d8 2 weeks ago   343MB
mysql                                8.0     e691422324d8 2 weeks ago   343MB
mysql                                latest  5709795eeffa 2 weeks ago   408MB
mysql/mysql-server                   latest  a3ee341faefb 5 weeks ago   246MB
store/oracle/mysql-enterprise-server 5.7     41bf2fa0b4a1 4 months ago  244MB

Run a MariaDB server container

Start a new Docker container from the MariaDB image by running:

shell> CONTAINER_NAME=mariadb
shell> CONTAINER_IMAGE=mariadb
shell> TAG=latest
shell> MYSQL_ROOT_PASSWORD=Secret-123
shell> MYSQL_ROOT_USER=root

shell> docker run \
  --name=${CONTAINER_NAME} \
  --detach \
  --env=MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} \
  ${CONTAINER_IMAGE}:${TAG}

shell> docker ps
CONTAINER ID IMAGE          COMMAND                CREATED        STATUS        PORTS    NAMES
60d7b6de7ed1 mariadb:latest "docker-entrypoint..." 24 seconds ago Up 23 seconds 3306/tcp mariadb

shell> docker logs ${CONTAINER_NAME}

shell> docker exec \
  --interactive \
  --tty \
  ${CONTAINER_NAME} \
  mysql --user=${MYSQL_ROOT_USER} --password=${MYSQL_ROOT_PASSWORD} --execute="status"

shell> docker image tag mariadb:latest mariadb:10.2.10

shell> docker exec --interactive \
  --tty \
  ${CONTAINER_NAME} \
  bash 

shell> docker stop ${CONTAINER_NAME}
shell> docker rm ${CONTAINER_NAME}

Run a MySQL Community server container

shell> CONTAINER_NAME=mysql
shell> CONTAINER_IMAGE=mysql/mysql-server
shell> TAG=latest
shell> MYSQL_ROOT_PASSWORD=Secret-123

shell> docker run \
  --name=${CONTAINER_NAME} \
  --detach \
  --env=MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} \
  ${CONTAINER_IMAGE}:${TAG}

shell> docker stop ${CONTAINER_NAME}
shell> docker rm ${CONTAINER_NAME}

Run a MySQL Server Enterprise Edition container

shell> CONTAINER_NAME=mysql-ee
shell> CONTAINER_IMAGE=store/oracle/mysql-enterprise-server
shell> TAG=5.7
shell> MYSQL_ROOT_PASSWORD=Secret-123

shell> docker run \
  --name=${CONTAINER_NAME} \
  --detach \
  --env=MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} \
  ${CONTAINER_IMAGE}:${TAG}

shell> docker ps --all
CONTAINER ID IMAGE                                    COMMAND                CREATED        STATUS                  PORTS               NAMES
0cb4e6a8a621 store/oracle/mysql-enterprise-server:5.7 "/entrypoint.sh my..." 37 seconds ago Up 36 seconds (healthy) 3306/tcp, 33060/tcp mysql-ee
1832b98da6ef mysql:latest                             "docker-entrypoint..." 6 minutes ago  Up 6 minutes            3306/tcp            mysql
60d7b6de7ed1 mariadb:latest                           "docker-entrypoint..." 21 minutes ago Up 21 minutes           3306/tcp            mariadb

All my 3 docker containers are currently running as root:

shell> ps -ef | grep docker
root 13177     1 20:20 ? 00:00:44 /usr/bin/dockerd -H fd://
root 13186 13177 20:20 ? 00:00:04 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 24004 13186 21:41 ? 00:00:00 docker-containerd-shim 60d7b6de7ed1ff62b67e66c6effce0094fd60e9565ede65fa34e188b636c54ec /var/run/docker/libcontainerd/60d7b6de7ed1ff62b67e66c6effce0094fd60e9565ede65fa34e188b636c54ec docker-runc
root 26593 13186 21:56 ? 00:00:00 docker-containerd-shim 1832b98da6ef7459c33181e9b9ddd89a4136c3b2676335bcbbb533389cbf6219 /var/run/docker/libcontainerd/1832b98da6ef7459c33181e9b9ddd89a4136c3b2676335bcbbb533389cbf6219 docker-runc
root 27714 13186 22:02 ? 00:00:00 docker-containerd-shim 0cb4e6a8a62103b66164ccddd028217bb4012d8a6aad1f62d3ed6ae71e1a38b4 /var/run/docker/libcontainerd/0cb4e6a8a62103b66164ccddd028217bb4012d8a6aad1f62d3ed6ae71e1a38b4 docker-runc

But the user running the process IN the container is not root:

shell> docker exec \
  --interactive \
  --tty \
  ${CONTAINER_NAME} \
  grep ^Uid /proc/1/status
Uid:    27      27      27      27

shell> docker exec \
  --interactive \
  --tty \
  ${CONTAINER_NAME} \
  bash -c "id 27"
uid=27(mysql) gid=27(mysql) groups=27(mysql)

Run a Docker container from mysql user

shell> id
uid=1001(mysql) gid=1001(mysql) groups=1001(mysql)
shell> docker images
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.32/images/json: dial unix /var/run/docker.sock: connect: permission denied

shell> sudo adduser mysql docker
Adding user `mysql' to group `docker' ...
Adding user mysql to group docker
Done.

Taxonomy upgrade extras: 

Galera Cluster and Antivirus Scanner on Linux

$
0
0

Today we had to investigate in a very strange behaviour of IST and SST on a MariaDB Galera Cluster.

The symptom was, that some Galera Cluster nodes took a very long time to start. Up to 7 minutes. So the customer was concluding that the Galera Cluster node does an SST instead of an IST and was asking why the SST happens.

It have to be mentioned here, that the MariaDB error log is very confusing about whether it is an SST or an IST. So the customer was confused and concluded, that MariaDB Galera Cluster was doing an SST instead of IST.

Further confusing was that this behaviour was not consistently on all 3 nodes and not consistently on the 3 stages production, test and integration.

First we had to clear if the Galera node was doing an IST or an SST to exclude problems with Galera Cache or event Bugs in MariaDB Galera Cluster. For this we were running our famous insert_test.sh and did some node restarts with forcing SST and without.

As a Galera Cluster operator you must mandatorily be capable to determine which one of both State Transfers happens from the MariaDB error log:

MariaDB Error Log with IST on Joiner

2017-12-12 22:29:33 140158145914624 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 204013)
2017-12-12 22:29:33 140158426741504 [Note] WSREP: State transfer required: 
        Group state: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:204013
        Local state: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:201439
2017-12-12 22:29:33 140158426741504 [Note] WSREP: New cluster view: global state: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:204013, view# 7: Primary, number of nodes: 3, my index: 2, protocol version 3
2017-12-12 22:29:33 140158426741504 [Warning] WSREP: Gap in state sequence. Need state transfer.
2017-12-12 22:29:33 140158116558592 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '127.0.0.1' --datadir '/home/mysql/database/magal-101-b/data/'  --defaults-file '/home/mysql/database/magal-101-b/etc/my.cnf'  --parent '16426' --binlog '/home/mysql/database/magal-101-b/binlog/laptop4_magal-101-b__binlog''
2017-12-12 22:29:33 140158426741504 [Note] WSREP: Prepared SST request: rsync|127.0.0.1:4444/rsync_sst
2017-12-12 22:29:33 140158426741504 [Note] WSREP: REPL Protocols: 7 (3, 2)
2017-12-12 22:29:33 140158426741504 [Note] WSREP: Assign initial position for certification: 204013, protocol version: 3
2017-12-12 22:29:33 140158203852544 [Note] WSREP: Service thread queue flushed.
2017-12-12 22:29:33 140158426741504 [Note] WSREP: IST receiver addr using tcp://127.0.0.1:5681
2017-12-12 22:29:33 140158426741504 [Note] WSREP: Prepared IST receiver, listening at: tcp://127.0.0.1:5681
2017-12-12 22:29:33 140158145914624 [Note] WSREP: Member 2.0 (Node B) requested state transfer from 'Node C'. Selected 1.0 (Node C)(SYNCED) as donor.
2017-12-12 22:29:33 140158145914624 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 204050)
2017-12-12 22:29:33 140158426741504 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-12-12 22:29:33 140158426741504 [Note] WSREP: GCache history reset: old(e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:0) -> new(e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:204013)
2017-12-12 22:29:33 140158145914624 [Note] WSREP: 1.0 (Node C): State transfer to 2.0 (Node B) complete.
2017-12-12 22:29:33 140158145914624 [Note] WSREP: Member 1.0 (Node C) synced with group.
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 16663 (20171212 22:29:34.474)
WSREP_SST: [INFO] Joiner cleanup done. (20171212 22:29:34.980)
2017-12-12 22:29:34 140158427056064 [Note] WSREP: SST complete, seqno: 201439
2017-12-12 22:29:35 140158427056064 [Note] WSREP: Signalling provider to continue.
2017-12-12 22:29:35 140158427056064 [Note] WSREP: SST received: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:201439
2017-12-12 22:29:35 140158426741504 [Note] WSREP: Receiving IST: 2574 writesets, seqnos 201439-204013
2017-12-12 22:29:35 140158426741504 [Note] WSREP: IST received: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:204013
2017-12-12 22:29:35 140158145914624 [Note] WSREP: 2.0 (Node B): State transfer from 1.0 (Node C) complete.
2017-12-12 22:29:35 140158145914624 [Note] WSREP: Shifting JOINER -> JOINED (TO: 204534)
2017-12-12 22:29:35 140158145914624 [Note] WSREP: Member 2.0 (Node B) synced with group.
2017-12-12 22:29:35 140158145914624 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 204535)
2017-12-12 22:29:35 140158426741504 [Note] WSREP: Synchronized with group, ready for connections

MariaDB Error Log with SST on Joiner

2017-12-12 22:32:15 139817123833600 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 239097)
2017-12-12 22:32:15 139817401395968 [Note] WSREP: State transfer required: 
        Group state: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:239097
        Local state: 00000000-0000-0000-0000-000000000000:-1
2017-12-12 22:32:15 139817401395968 [Note] WSREP: New cluster view: global state: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:239097, view# 9: Primary, number of nodes: 3, my index: 2, protocol version 3
2017-12-12 22:32:15 139817401395968 [Warning] WSREP: Gap in state sequence. Need state transfer.
2017-12-12 22:32:15 139817094477568 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '127.0.0.1' --datadir '/home/mysql/database/magal-101-b/data/'  --defaults-file '/home/mysql/database/magal-101-b/etc/my.cnf'  --parent '25291' --binlog '/home/mysql/database/magal-101-b/binlog/laptop4_magal-101-b__binlog''
2017-12-12 22:32:15 139817401395968 [Note] WSREP: Prepared SST request: rsync|127.0.0.1:4444/rsync_sst
2017-12-12 22:32:15 139817401395968 [Note] WSREP: REPL Protocols: 7 (3, 2)
2017-12-12 22:32:15 139817401395968 [Note] WSREP: Assign initial position for certification: 239097, protocol version: 3
2017-12-12 22:32:15 139817178507008 [Note] WSREP: Service thread queue flushed.
2017-12-12 22:32:15 139817401395968 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (e2fbbca5-df26-11e7-8ee2-bb61f8ff3774): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2017-12-12 22:32:15 139817123833600 [Note] WSREP: Member 2.0 (Node B) requested state transfer from 'Node C'. Selected 1.0 (Node C)(SYNCED) as donor.
2017-12-12 22:32:15 139817123833600 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 239136)
2017-12-12 22:32:15 139817401395968 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-12-12 22:32:15 139817401395968 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:239097)
2017-12-12 22:32:17 139817123833600 [Note] WSREP: 1.0 (Node C): State transfer to 2.0 (Node B) complete.
2017-12-12 22:32:17 139817123833600 [Note] WSREP: Member 1.0 (Node C) synced with group.
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 25520 (20171212 22:32:17.846)
WSREP_SST: [INFO] Joiner cleanup done. (20171212 22:32:18.352)
2017-12-12 22:32:18 139817401710528 [Note] WSREP: SST complete, seqno: 239153
2017-12-12 22:32:18 139817132226304 [Note] WSREP: (ebfd9e9c, 'tcp://127.0.0.1:5680') turning message relay requesting off
2017-12-12 22:32:22 139817401710528 [Note] WSREP: Signalling provider to continue.
2017-12-12 22:32:22 139817401710528 [Note] WSREP: SST received: e2fbbca5-df26-11e7-8ee2-bb61f8ff3774:239153
2017-12-12 22:32:22 139817123833600 [Note] WSREP: 2.0 (Node B): State transfer from 1.0 (Node C) complete.
2017-12-12 22:32:22 139817123833600 [Note] WSREP: Shifting JOINER -> JOINED (TO: 239858)
2017-12-12 22:32:22 139817123833600 [Note] WSREP: Member 2.0 (Node B) synced with group.
2017-12-12 22:32:22 139817123833600 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 239866)
2017-12-12 22:32:22 139817401395968 [Note] WSREP: Synchronized with group, ready for connections

After we cleared that it really was an IST and that it was not a SST because of some other reasons the question rose: Why does an IST of only a few thousand transactions was taking 420 seconds. And this was not always the case...

So we were looking with top at the Donor and the Joiner during IST and we found that on the Donor node the Antivirus software was heavily using CPU (2 x 50%) and otherwise the system was doing nothing for a while and then suddenly started to transfer data over the network (possibly IST?).
Later we found, that the MariaDB datadir (/var/lib/mysql) was not excluded from the Antivirus software. And finally it looks like the Antivirus software was not properly configured by its Master server because the Antivirus software agent was from a cloned VM and not reinitialized. So the Antivirus Master server seems to be confused because there are 2 Antivirus software agents with the same ID.

Another very surprising situation which we did not expect was, that IST was much heavier influenced by the Antivirus software than SST. SST finished in a few seconds while IST took 420 seconds.

Conclusion: Be careful when using Antivirus software in combination with MariaDB Galera Cluster databases and exclude at least all database directories from virus scanning. If you want to be sure to avoid side effects (noisy neighbours) disable the Antivirus software on the database server at all and make sure by other means, that no virus is reaching your precious MariaDB Galera Cluster...

Oracle releases MySQL security vulnerability fixes 2018-01

$
0
0

As in every quarter of the year Oracle has released yesterday its recommendation for the MySQL security updates. This is called, in Oracle terminology, Critical Patch Update (CPU) Advisory.

This CPU is published for all Oracle products. But FromDual is only interested in MySQL related topics. So let us concentrate on those.

This time 25 fixes with a maximum score of 8.1 (out of 10.0) were published.

6 of theses 25 vulnerabilities are exploitable remotely over the network without authentication (no user credentials required)!

The following MySQL products are affected:

  • MySQL Enterprise Monitor (3.3.6.3293 and before, 3.4.4.4226 and before, 4.0.0.5135 and before)
  • MySQL Connector/Net (6.9.9. and before, 6.10.4 and before)
  • MySQL Connector/ODBC (5.3.9. and before)
  • MySQL Server (5.5.58 and before, 5.6.38 and before, 5.7.19 and before)

It is recommended to upgrade your MySQL products to close the security vulnerabilities.

FromDual upgrade decision aid

Because such security updates are published quarterly and some of our customers have dozens to hundreds of MySQL installations this would end up in a never ending story where you are continuously upgrading MySQL database servers and other products.

This led to idea to create an upgrade decision aid to decide if you have to upgrade to this CPU or not.

The following questions can be asked:

  • How exposed is your database?
    Databases can be located in various network segments. It is not recommended to expose databases directly to the internet. Databases are either installed in demilitarized zones (DMZ) with no direct access from the internet or in the companies private network (only company employees should be able to access the database) or even specialized secure networks (only a limited number of specific employees can access this network).
  • How critical are your data?
    Some data are more interesting or critical, some data are less interesting or critical. Interesting data are: User data (user name and password), customer data (profiles, preferences, etc.), financial data (credit cards) and health care data (medical data). Systems containing such data are more critical than others. You can also ask: How sever is it if such data leak?
  • How broad is the user base able to access the database?
    How many employees do you have in your company? How many contractors do you have in your company? How many employees have physical access to the database server? How good is the mood of those people?
    How good are the user credentials to protect your database? Do you have shared passwords or no passwords at all? Do you have an account management (expiring old accounts, rotate passwords from time to time)?
    How much do you trust your users? Do you trust all your employees? Do you trust only admins? Or do you not even trust your admins?
  • How severe are the security vulnerabilities?
    You can define a threshold of severity of the vulnerabilities above you want to take actions. According to your criticality you can take actions for example as follows: Greater or equal than 7.5 if you have less critical data. Greater or equal than 6.0 if you have critical data.
  • Can the vulnerability be use from remote (over the network) and does it need a user authentication to exploit the vulnerability? What products (MySQL Enterprise Monitor, MySQL Server, MySQL Connectors) and what modules (Apache/Tomcat, .Net Connector, Partitioning, Stored Procedures, InnoDB, DDL, GIS, Optimizer, ODBC, Replication, DML, Performance Schema) are affected?

Depending on your readiness to take a risk you get now answers to decide if you have to take actions or not.

Some examples

  • Situation: Your database is exposed directly to the internet or you forgot to install some firewall rules to protect your MySQL port.
    Analysis: You are probably affected by CVE-2018-2696 and CVE-2017-3737 (score 5.9 and 7.5). So you passed the threshold for non-critical data (7.5) and nearly passed the threshold for critical data (6.0). These vulnerabilities allow attacks over the network without user authentication.
    Action: Immediate upgrade is recommended. Mid-term action: Install firewall rules to protect your MySQL to avoid access from remote and/or do not expose databases directly to the internet.
  • Situation: Your database is located in the intranet zone. You have slack user/password policies and you have many employees and also many contractors from foreign countries working on various projects. And you have very sensitive/interesting financial data stored in your database.
    Analysis: Many people, not all of them are really trusted, have network access to the database. It is quite possible that passwords have been shared or people have passwords for projects they are not working for any more. You are affected by nearly all of the vulnerabilities (network).
    Action: You should plan an upgrade soon. Mid-term action: Try to restrict access to the databases and implement some password policy rules (no shared passwords, password expiration, account locking etc.).
  • Situation: Your highly critical databases are located in a specially secured network and only applications, Linux admins and DBAs have access to this network. And you completely trust those people.
    Analysis: Your threshold is 6.0 and (unauthenticated) attack over the network is not possible. There are some vulnerabilities of which you are affected but the database is only accessed by an application. So those vulnerabilities cannot be exploited easily.
    Action: You possibly can ignore this CPU for the MySQL database this time. But you have a vulnerability in the .Net Connector (Connector/Net). If an attacker exploits the vulnerability on the Connector he possibly can get access to the data. So you have to upgrade the Connector of your application accessing the database.

If you follow the ideas of this aid you will probably have one or two upgrades a year. And this you should do anyway just to stay up to date...

See also Common Vulnerability Scoring System Version 3.0 Calculator.

Taxonomy upgrade extras: 

Advanced MySQL and MariaDB training in Cologne 2018

$
0
0

End of February, from February 26 to March 2 (5 days), FromDual offers an additional training for DBAs and DevOps: our most visited Advanced MySQL and MariaDB training.

This training is hold in the training facilities of the FromDual training partner GFU Cyrus GmbH in Cologne-Deutz (Germany).

There are already enough registrations so it is certain the training will take place. But there are still free places for at least 3 additional participants.

The training is in German.

You can find the training of this 5-day MySQL/MariaDB training here.

If you have any question please do not hesitate to contact us.

Taxonomy upgrade extras: 

Short term notice: Percona XtraDB Cluster training in English 7/8 February 2018 in Germany

$
0
0

FromDual offers short term a Percona XtraDB Cluster and MySQL Galera Cluster training (2 days) in English.

The training will take place in the Linuxhotel in Essen/Germany on February 7/8 2018.

There are already enough registrations so it is certain the training will take place. But there are still free places for some additional participants.

You can book online at the Linuxhotel.

Important: The Linuxhotel is nearly fully booked out. So accommodation is in nearby locations. The Linuxhotel will recommend you some locations.

The training is in English.

You can find the contents of this 2-day Percona XtraDB Cluster training here.

If you have any question please do not hesitate to contact us.

MySQL 8.0.4-rc is out

$
0
0

Yesterday MySQL 8.0.4-rc came out. The Release Notes are quite long.
But caution: Do a BACKUP before upgrading...

I experienced some nice surprises. First I have to admit that I did not read the Release Notes or anything else. Reading manuals is for Girlies! Possibly something is written in there which is of importance. But I expect that it just works as usual...

I downloaded MySQL 8.0.4-rc and just want to upgrade my MySQL 8.0.3-rc testing system, where we did the 1M tables test.

First I got:

[MY-011096] No data dictionary version number found.
[MY-010020] Data Dictionary initialization failed.
[MY-010119] Aborting

Hmmm... Maybe something was not clean with the old system. So downgrade again:

[ERROR] [000000] InnoDB: Unsupported redo log format. The redo log was created with MySQL 8.0.4. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading-downgrading.html
[ERROR] [000000] InnoDB: Plugin initialization aborted with error Generic error
[ERROR] [003957] Failed to initialize DD Storage Engine
[ERROR] [003634] Data Dictionary initialization failed.
[ERROR] [003742] Aborting

OK. Understandable. I should have done a backup before. But backup is for Girlies as well! Anyway this test system is not important. So I created a new instance from scratch which finally worked... Possibly just removing the redo log files as indicated would have helped as well.

Advanced MySQL Enterprise Training by FromDual

$
0
0

Due to the increasing demand FromDual has developed an Advanced MySQL Enterprise Training for DBAs and DevOps. After testing this training extensively with some selected customers last year we offer this MySQL Enterprise Training in 2018 for a broader audience.

The MySQL Enterprise Training addresses MySQL DBAs and DevOps which are already familiar with MySQL and approach now the challenge to operate a serious MySQL Enterprise infrastructure.

The topics of the 3 days MySQL Enterprise training you can find here.

You further have the opportunity to add 2 extra days of MySQL Performance Tuning from the Advanced MySQL Training.

We would be pleased to hold this training in-house in your company or at the location of one of our training partners in Essen, Berlin and Cologne (Germany).

For any question please contact us by eMail.

Taxonomy upgrade extras: 

MySQL Environment MyEnv 2.0.0 has been released

$
0
0

FromDual has the pleasure to announce the release of the new version 2.0.0 of its popular MySQL, Galera Cluster and MariaDB multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to the FromDual bug tracker.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 1.1.x to 2.0.0

MyEnv 2.0.0 requires an new PHP package for socket handling. On Red Hat, CentOS, Debian and Ubuntu it seems to be installed by default. On OpenSUSE it has to be installed (php-sockets). For more details see also our MyEnv Installation Guide.

shell> sudo zypper install php-sockets   # on OpenSUSE and SLES only

shell> cd ${HOME}/product
shell> tar xf /download/myenv-2.0.0.tar.gz
shell> rm -f myenv
shell> ln -s myenv-2.0.0 myenv

Plug-ins

If you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:

shell> cd ${HOME}/product/myenv
shell> ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/

Upgrade of the instance directory structure

From MyEnv v1 to v2 the directory structure of instances has fundamentally changed. Nevertheless MyEnv v2 works fine with MyEnv v1 directory structures.

Old structure

~/data/instance1/ibdata1
~/data/instance1/ib_logfile?
~/data/instance1/my.cnf
~/data/instance1/error.log
~/data/instance1/mysql
~/data/instance1/test~/data/mypprod/
~/data/instance1/general.log
~/data/instance1/slow.log
~/data/instance1/binlog.0000??
~/data/instance2/...

New structure

~/database/instance1/binlog/binlog.0000??
~/database/instance1/data/ibdata1
~/database/instance1/data/ib_logfile?
~/database/instance1/data/mysql
~/database/instance1/data/test
~/database/instance1/etc/my.cnf
~/database/instance1/log/error.log
~/database/instance1/log/general.log
~/database/instance1/log/slow.log
~/database/instance1/tmp/
~/database/instance2/...

But over time you possibly want to migrate the old structure to the new one. The following steps describe how you upgrade MyEnv instance structure v1 to v2:

mysql@chef:~ [mysql-57, 3320]> mypprod
mysql@chef:~ [mypprod, 3309]> stop
.. SUCCESS!
mysql@chef:~ [mypprod, 3309]> mkdir ~/database/mypprod
mysql@chef:~ [mypprod, 3309]> mkdir ~/database/mypprod/binlog ~/database/mypprod/data ~/database/mypprod/etc ~/database/mypprod/log ~/database/mypprod/tmp
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/binary-log.* ~/database/mypprod/binlog/
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/my.cnf ~/database/mypprod/etc/
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/error.log ~/database/mypprod/log/
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/slow.log ~/database/mypprod/log/
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/general.log ~/database/mypprod/log/
mysql@chef:~ [mypprod, 3309]> mv ~/data/mypprod/* ~/database/mypprod/data/
mysql@chef:~ [mypprod, 3309]> rmdir ~/data/mypprod
mysql@chef:~ [mypprod, 3309]> vi /etc/myenv/myenv.conf

- datadir              = /home/mysql/data/mypprod
+ datadir              = /home/mysql/database/mypprod/data
- my.cnf               = /home/mysql/data/mypprod/my.cnf
+ my.cnf               = /home/mysql/database/mypprod/etc/my.cnf
+ instancedir          = /home/mysql/database/mypprod

mysql@chef:~ [mypprod, 3309]> source ~/.bash_profile
mysql@chef:~ [mypprod, 3309]> cde
mysql@chef:~/database/mypprod/etc [mypprod, 3309]> vi my.cnf 

- log_bin                                = binary-log
+ log_bin                                = /home/mysql/database/mypprod/binlog/binary-log
- datadir                                = /home/mysql/data/mypprod
+ datadir                                = /home/mysql/database/mypprod/data
- tmpdir                                 = /tmp
+ tmpdir                                 = /home/mysql/database/mypprod/tmp
- log_error                              = error.log
+ log_error                              = /home/mysql/database/mypprod/log/error.log
- slow_query_log_file                    = slow.log
+ slow_query_log_file                    = /home/mysql/database/mypprod/log/slow.log
- general_log_file                       = general.log
+ general_log_file                       = /home/mysql/database/mypprod/log/general.log

mysql@chef:~/database/mypprod/etc [mypprod, 3309]> cdb
mysql@chef:~/database/mypprod/binlog [mypprod, 3309]> vi binary-log.index 

- ./binary-log.000001
+ /home/mysql/database/mypprod/binlog/binary-log.000001
- ./binary-log.000001
+ /home/mysql/database/mypprod/binlog/binary-log.000001

mysql@chef:~/database/mypprod/binlog [mypprod, 3309]> start
mysql@chef:~/database/mypprod/binlog [mypprod, 3309]> exit

Changes in MyEnv 2.0.0

MyEnv

  • New v2 instance directory structure and instancedir variable introduced, aliases adapted accordingly.
  • Configuration files aliases.conf and variables.conf made more user friendly.
  • PHP 7 support added.
  • Made MyEnv MySQL 8.0 ready.
  • Packaging (DEB/RPM) for RHEL 6 and 7 and SLES 11 and 12 DEB (Ubuntu/Debian) available.
  • OEM agent plug-in made ready for OEM v12.
  • More strict configuration checking.
  • Version more verbose.
  • Database health check mysqladmin replace by UNIX socket probing.
  • Various bug fixes (#168, #161, ...)
  • MyEnv made ready for systemd.
  • Bind-address output nicer in up.
  • New variables added to my.cnf template (super_read_only, innodb_tmpdir, innodb_flush_log_at_trx_commit, MySQL Group Replication, crash-safe Replication, GTID, MySQL 8.0)

MyEnv Installer

  • Installer made ready for systemd.
  • Question for angel process (mysqld_safe) and cgroups added.
  • Check for duplicate socket added.
  • Various bug fixes.
  • Purge data implemented.

MyEnv Utilities

  • Utility mysqlstat.php added.
  • Scripts for keepalived added.
  • Utilities mysql-create-instance.sh and mysql-remove-instance.sh removed.
  • Famous insert_test.sh, insert_test.php and test table improved.

For subscriptions of commercial use of MyEnv please get in contact with us.


MariaDB 10.2 New Features - Slides available

$
0
0

The Chemnitz Linux Days 2018 in Chemnitz (Germany) 10/11 March 2018 are over for more than a week now and IMHO it was a huge success.

I was following many very interesting talks, met a lot of interesting people and learned a lot!

For all those who could not follow our presentation about New Features in MariaDB 10.2 (PDF, 683 kib) the presentation slides are on-line available.

If you want to hear the presentation live you can join us at the SLAC 2018 in Berlin.

MySQL sys Schema in MariaDB 10.2

$
0
0

MySQL has introduced the PERFORMANCE_SCHEMA (P_S) in MySQL 5.5 and made it really usable in MySQL 5.6 and added some enhancements in MySQL 5.7 and 8.0.

Unfortunately the PERFORMANCE_SCHEMA was not really intuitive for the broader audience. Thus Mark Leith created the sys Schema for an easier access for the normal DBA and DevOps and Daniel Fischer has enhanced it further. Fortunately the sys Schema up to version 1.5.1 is available on GitHub. So we can adapt and use it for MariaDB as well. The version of the sys Schema in MySQL 8.0 is 1.6.0 and seems not to be on GitHub yet. But you can extract it from the MySQL 8.0 directory structure: mysql-8.0/share/mysql_sys_schema.sql. According to a well informed source the project on GitHub is not dead but the developers have just been working on other priorities. An the source announced another release soon (they are working on it at the moment).

MariaDB has integrated the PERFORMANCE_SCHEMA based on MySQL 5.6 into its own MariaDB 10.2 server but unfortunately did not integrate the sys Schema. Which PERFORMANCE_SCHEMA version is integrated in MariaDB can be found here.

To install the sys Schema into MariaDB we first have to check if the PERFORMANCE_SCHEMA is activated in the MariaDB server:

mariadb> SHOW GLOBAL VARIABLES LIKE 'performance_schema';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| performance_schema | OFF   |
+--------------------+-------+

To enable the PERFORMANCE_SCHEMA just add the following line to your my.cnf:

[mysqld]

performance_schema = 1

and restart the instance.

In MariaDB 10.2 the MySQL 5.6 PERFORMANCE_SCHEMA is integrated so we have to run the sys_56.sql installation script. If you try to run the sys_57.sql script you will get a lot of errors...

But also the sys_56.sql installation script will cause you some little troubles which are easy to fix:

unzip mysql-sys-1.5.1.zip 
mysql -uroot < sys_56.sql

ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'server_uuid'
ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'master_info_repository'
ERROR 1193 (HY000) at line 20 in file: './procedures/diagnostics.sql': Unknown system variable 'relay_log_info_repository'

For a quick hack to make the sys Schema work I changed the following information:

  • server_uuid to server_id
  • @@master_info_repository to NULL (3 times).
  • @@relay_log_info_repository to NULL (3 times).

For the future the community has to think about if the sys Schema should be aware of the 2 branches MariaDB and MySQL and act accordingly or if the sys Schema has to be forked to work properly for MariaDB and implement MariaDB specific functionality.

When the sys Schema finally is installed you have the following tables to get your performance metrics:

mariadb> use sys
mariadb> SHOW TABLES;
+-----------------------------------------------+
| Tables_in_sys                                 |
+-----------------------------------------------+
| host_summary                                  |
| host_summary_by_file_io                       |
| host_summary_by_file_io_type                  |
| host_summary_by_stages                        |
| host_summary_by_statement_latency             |
| host_summary_by_statement_type                |
| innodb_buffer_stats_by_schema                 |
| innodb_buffer_stats_by_table                  |
| innodb_lock_waits                             |
| io_by_thread_by_latency                       |
| io_global_by_file_by_bytes                    |
| io_global_by_file_by_latency                  |
| io_global_by_wait_by_bytes                    |
| io_global_by_wait_by_latency                  |
| latest_file_io                                |
| metrics                                       |
| processlist                                   |
| ps_check_lost_instrumentation                 |
| schema_auto_increment_columns                 |
| schema_index_statistics                       |
| schema_object_overview                        |
| schema_redundant_indexes                      |
| schema_table_statistics                       |
| schema_table_statistics_with_buffer           |
| schema_tables_with_full_table_scans           |
| schema_unused_indexes                         |
| session                                       |
| statement_analysis                            |
| statements_with_errors_or_warnings            |
| statements_with_full_table_scans              |
| statements_with_runtimes_in_95th_percentile   |
| statements_with_sorting                       |
| statements_with_temp_tables                   |
| sys_config                                    |
| user_summary                                  |
| user_summary_by_file_io                       |
| user_summary_by_file_io_type                  |
| user_summary_by_stages                        |
| user_summary_by_statement_latency             |
| user_summary_by_statement_type                |
| version                                       |
| wait_classes_global_by_avg_latency            |
| wait_classes_global_by_latency                |
| waits_by_host_by_latency                      |
| waits_by_user_by_latency                      |
| waits_global_by_latency                       |
+-----------------------------------------------+

One query as an example: Top 10 MariaDB global I/O latency files on my system:

mariadb> SELECT * FROM sys.waits_global_by_latency LIMIT 10;
+--------------------------------------+-------+---------------+-------------+-------------+
| events                               | total | total_latency | avg_latency | max_latency |
+--------------------------------------+-------+---------------+-------------+-------------+
| wait/io/file/innodb/innodb_log_file  |   112 | 674.18 ms     | 6.02 ms     | 23.75 ms    |
| wait/io/file/innodb/innodb_data_file |   892 | 394.60 ms     | 442.38 us   | 29.74 ms    |
| wait/io/file/sql/FRM                 |   668 | 72.85 ms      | 109.05 us   | 20.17 ms    |
| wait/io/file/sql/binlog_index        |    10 | 21.25 ms      | 2.13 ms     | 15.74 ms    |
| wait/io/file/sql/binlog              |    19 | 11.18 ms      | 588.56 us   | 10.38 ms    |
| wait/io/file/myisam/dfile            |    79 | 10.48 ms      | 132.66 us   | 3.78 ms     |
| wait/io/file/myisam/kfile            |    86 | 7.23 ms       | 84.01 us    | 789.44 us   |
| wait/io/file/sql/dbopt               |    35 | 1.95 ms       | 55.61 us    | 821.68 us   |
| wait/io/file/aria/MAI                |   269 | 1.18 ms       | 4.40 us     | 91.20 us    |
| wait/io/table/sql/handler            |    36 | 710.89 us     | 19.75 us    | 125.37 us   |
+--------------------------------------+-------+---------------+-------------+-------------+

Taxonomy upgrade extras: 

Special MySQL and MariaDB trainings 2018 in English

$
0
0

Due to a strong customer demand FromDual offers 2018 two extra MySQL/MariaDB trainings with its Training partner The Linuxhotel in Essen (Germany). Those trainings are in English.

  • MariaDB Performance Tuning on 5 and 6 September 2018 (2 days).
  • Advanced MySQL/MariaDB training on 26 to 30 November 2018 (5 days).

More information about the contents of the trainings can be found at Advanced MySQL and MariaDB training.

For conditions and booking: MariaDB Performance Tuning and Advanced MySQL Training.

For specific MariaDB or MySQL on-site Consulting or in-house Training please get in contact with us.

Select Hello World FromDual with MariaDB PL/SQL

$
0
0

MariaDB 10.3 was released GA a few weeks ago. One of the features which interests me most is the MariaDB Oracle PL/SQL compatibility mode.

So its time to try it out now...

Enabling Oracle PL/SQL in MariaDB

Oracle PL/SQL syntax is quite different from old MySQL/MariaDB SQL/PSM syntax. So the old MariaDB parser would through some errors without modification. The activation of the modification of the MariaDB PL/SQL parser is achieved by changing the sql_mode as follows:

mariadb> SET SESSION sql_mode=ORACLE;

or you can make this setting persistent in your my.cnf MariaDB configuration file:

[mysqld]

sql_mode = ORACLE

To verify if the sql_mode is already set you can use the following statement:

mariadb> pager grep --color -i oracle
PAGER set to 'grep --color -i oracle'
mariadb> SELECT @@sql_mode;
| PIPES_AS_CONCAT,ANSI_QUOTES,IGNORE_SPACE,ORACLE,NO_KEY_OPTIONS,NO_TABLE_OPTIONS,NO_FIELD_OPTIONS,NO_AUTO_CREATE_USER,SIMULTANEOUS_ASSIGNMENT |
mariadb> nopager

Nomen est omen

First of all I tried the function of the basic and fundamental table in Oracle, the DUAL table:

mariadb> SELECT * FROM dual;
ERROR 1096 (HY000): No tables used

Sad. :-( But this query on the dual table seems to work:

mariadb> SELECT 'Hello World!' FROM dual;
+--------------+
| Hello World! |
+--------------+
| Hello World! |
+--------------+

The second result looks much better. The first query should work as well but does not. We opened a bug at MariaDB without much hope that this bug will be fixed soon...

To get more info why MariaDB behaves like this I tried to investigate a bit more:

mariadb> SELECT table_schema, table_name
  FROM information_schema.tables
 WHERE table_name = 'dual';
Empty set (0.001 sec)

Hmmm. It seems to be implemented not as a real table... But normal usage of this table seems to work:

mariadb> SELECT CURRENT_TIMESTAMP() FROM dual;
+---------------------+
| current_timestamp() |
+---------------------+
| 2018-06-07 15:32:11 |
+---------------------+

If you rely heavily in your code on the dual table you can create it yourself. It is defined as follows:

"The DUAL table has one column, DUMMY, defined to be VARCHAR2(1), and contains one row with a value X."

If you want to create the dual table yourself here is the statement:

mariadb> CREATE TABLE `DUAL` (DUMMY VARCHAR2(1));
mariadb> INSERT INTO `DUAL` (DUMMY) VALUES ('X');

Anonymous PL/SQL block in MariaDB

To try some PL/SQL features out or to run a sequence of PL/SQL commands you can use anonymous blocks. Unfortunately MySQL SQL/PSM style delimiter seems still to be necessary.

It is recommended to use the DELIMITER /, then most of the Oracle examples will work straight out of the box...

DELIMITER /

BEGIN
  SELECT 'Hello world from MariaDB anonymous PL/SQL block!';
END;
/

DELIMITER ;

+--------------------------------------------------+
| Hello world from MariaDB anonymous PL/SQL block! |
+--------------------------------------------------+
| Hello world from MariaDB anonymous PL/SQL block! |
+--------------------------------------------------+

A simple PL/SQL style MariaDB Procedure

DELIMITER /

CREATE OR REPLACE PROCEDURE hello AS
BEGIN
  DECLARE
    vString VARCHAR2(255) := NULL;
  BEGIN
    SELECT 'Hello world from MariaDB PL/SQL Procedure!' INTO vString FROM dual;
    SELECT vString;
  END;
END hello;
/

BEGIN
  hello();
END;
/

DELIMITER ;

A simple PL/SQL style MariaDB Function

DELIMITER /

CREATE OR REPLACE FUNCTION hello RETURN VARCHAR2 DETERMINISTIC AS
BEGIN
  DECLARE
    vString VARCHAR2(255) := NULL;
  BEGIN
    SELECT 'Hello world from MariaDB PL/SQL Function!' INTO vString FROM dual;
    RETURN vString;
  END;
END hello;
/

DECLARE
  vString VARCHAR(255) := NULL;
BEGIN
  vString := hello();
  SELECT vString;
END;
/

DELIMITER ;

An PL/SQL package in MariaDB

Up to here there is nothing really new, just slightly different. But now let us try a PL/SQL package in MariaDB:

DELIMITER /

CREATE OR REPLACE PACKAGE hello AS
  -- must be delared as public!
  PROCEDURE helloWorldProcedure(pString VARCHAR2);
  FUNCTION helloWorldFunction(pString VARCHAR2) RETURN VARCHAR2;
END hello;
/

CREATE OR REPLACE PACKAGE BODY hello AS

  vString VARCHAR2(255) := NULL;

  -- was declared public in PACKAGE
  PROCEDURE helloWorldProcedure(pString VARCHAR2) AS
  BEGIN
    SELECT 'Hello world from MariaDB Package Procedure in ' || pString || '!' INTO vString FROM dual;
    SELECT vString;
  END;

  -- was declared public in PACKAGE
  FUNCTION helloWorldFunction(pString VARCHAR2) RETURN VARCHAR2 AS
  BEGIN
    SELECT 'Hello world from MariaDB Package Function in ' || pString || '!' INTO vString FROM dual;
    return vString;
  END;
BEGIN
  SELECT 'Package initialiser, called only once per connection!';
END hello;
/

DECLARE
  vString VARCHAR2(255) := NULL;
  -- CONSTANT seems to be not supported yet by MariaDB
  -- cString CONSTANT VARCHAR2(255) := 'anonymous block';
  cString VARCHAR2(255) := 'anonymous block';
BEGIN
  CALL hello.helloWorldProcedure(cString);
  SELECT hello.helloWorldFunction(cString) INTO vString;
  SELECT vString;
END;
/

DELIMITER ;

DBMS_OUTPUT package for MariaDB

An Oracle database contains over 200 PL/SQL packages. One of the most common one is the DBMS_OUTPUT package. In this package we can find the Procedure PUT_LINE.

This package/function has not been implemented yet by MariaDB so far. So we have to do it ourself:

DELIMITER /

CREATE OR REPLACE PACKAGE DBMS_OUTPUT AS
  PROCEDURE PUT_LINE(pString IN VARCHAR2);
END DBMS_OUTPUT;
/

CREATE OR REPLACE PACKAGE BODY DBMS_OUTPUT AS

  PROCEDURE PUT_LINE(pString IN VARCHAR2) AS
  BEGIN
    SELECT pString;
  END;
END DBMS_OUTPUT;
/

BEGIN
  DBMS_OUTPUT.PUT_LINE('Hello world from MariaDB DBMS_OUTPUT.PUT_LINE!');
END;
/

DELIMITER ;

The other Functions and Procedures have to be implemented later over time...

Now we can try to do all examples from Oracle sources!

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.0.0 has been released

$
0
0

FromDual has the pleasure to announce the release of the new version 2.0.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. How to install and use the Backup and Recovery Manager is describe in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 1.2.x to 2.0.0

brman 2.0.0 requires a new PHP package for ssh connections.

shell> sudo apt-get install php-ssh2

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.0.0.tar.gz
shell> rm -f brman
shell> ln -s brman-2.0.0 brman

Changes in FromDual Backup and Recovery Manager 2.0.0

This release is a new major release series. It contains a lot of new features. We have tried to maintain backward-compatibility with the 1.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version

FromDual Backup Manager

  • brman was made ready for MariaDB 10.3
  • brman was made ready for MySQL 8.0
  • DEB and RPM packages are prepared.
  • fromdual_brman was renamed to brman (because of DEB).
  • mariabackup support was added.
  • Timestamp for physical backup and compression added to log file.
  • Option --no-purge added to not purge binary logs during binlog backup.
  • A failed archive command in physical full backup does not abort backup loop any more.
  • Return code detection for different physical backup tools improved.
  • Bug in fetching binlog file and position from xtrabackup_binlog_info fixed.
  • Version made MyEnv compliant.
  • Errors and warnings are written to STDERR.
  • General Tablespace check implemented. Bug in mysqldump. Only affects MySQL 5.7 and 8.0. MariaDB up to 10.3 has not implemented this feature yet.
  • Warning messages improved.
  • Option --quick added for logical (mysqldump) backup to speed up backup.
  • On schema backups FLUSH BINARY LOGS is executed only once when --per-schema backup is used.
  • The database user root should not be used for backups any more. User brman is suggested.
  • Option --pass-through is implemented to pass options like --ignore-table through to the backend backup tool (mysqldump, mariabackup, xtrabackup, mysqlbackup).
  • bman can report to fpmmm/Zabbix now.
  • Check for binary logging made less intrusive.
  • All return codes (rc) are matching to new schema now. That means errors do not necessarily have same error codes with new brman version.
  • If RELOAD privilege is missing --master-data and/or --flush-logs options are omitted. This makes bman backups possible for some shared hosting and cloud environments.
  • Schema backup does not require SHOW DATABASES privilege any more. This makes it possible to use bman for shared hosting and cloud environments.
  • Info messages made nicer with empty lines.
  • Option --archivedir is replaced by --archivedestination.
  • Remote copy of backup via rsync, scp and sftp is possible.
  • Connect string was shown wrong in the log file.
  • Connect string of target and catalog made URI conform.
  • bman supports now mariabackup, xtrabackup and mysqlbackup properly (recent releases).

FromDual Backup Manager Catalog

  • Catalog write is done if physical backup hits an error in archiving.
  • Renamed catalog to brman_catalog.

For subscriptions of commercial use of brman please get in contact with us.

Cool new features in FromDual Backup and Recovery Manager 2.0.0

$
0
0

A while ago we released our FromDual Backup and Recovery Manager (brman) 2.0.0 for MariaDB and MySQL. So what are the new cool features of this new release?

First of all brman 2.0.0 is compatible with MariaDB 10.3 and MySQL 8.0:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --mode=logical --policy=daily

Reading configuration from /etc/mysql/my.cnf
Reading configuration from /home/mysql/.my.cnf
No bman configuration file.

Command line: /home/mysql/product/brman-2.0.0/bin/bman.php --target=brman:******@127.0.0.1:3318 --type=full --mode=logical --policy=daily 

Options from command line
  target          = brman:******@127.0.0.1:3318
  type            = full
  mode            = logical
  policy          = daily

Resulting options
  config          = 
  target          = brman:******@127.0.0.1:3318
  type            = full
  mode            = logical
  policy          = daily
  log             = ./bman.log
  backupdir       = /home/mysql/bck
  catalog-name    = brman_catalog

Logging to   ./bman.log
Backupdir is /home/mysql/bck
Hostname is  chef
Version is   2.0.0 (catalog v0.2.0)

Start backup at 2018-08-13_11-57-31
  Binary logging is disabled.
  Schema to backup: mysql, foodmart, world, test

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  foodmart                    0             0             0           0
  mysql            CSV        2             0             0           4
  mysql            InnoDB     4         65536         49152          17
  mysql            MyISAM    25        515327        133120        2052
  test             InnoDB     3         49152             0           0
  world                       0             0             0           0

  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318 --all-databases  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --events
  to Destination: /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  Backup size is 488835
  Backup does NOT contain any binary log information.
  Do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
  md5 = 31cab19021e01c12db5fe49165a3df93
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_11-57-31.sql
End backup at 2018-08-13 11:57:31 (rc=0)

Next brman also support mariabackup now:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --mode=physical --policy=daily
...
Start backup at 2018-08-13_12-02-18
  Backup with tool mariabackup version 10.3.7 (from path /home/mysql/product/mariadb-10.3/bin/mariabackup).
  Schema to backup: mysql, foodmart, world, test

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  foodmart                    0             0             0           0
  mysql            CSV        2             0             0           4
  mysql            InnoDB     4         65536         49152          17
  mysql            MyISAM    25        515327        133120        2052
  test             InnoDB     3         49152             0           0
  world                       0             0             0           0

  Binary logging is disabled.
  /home/mysql/product/mariadb-10.3/bin/mariabackup --defaults-file=/tmp/bck_full_2018-08-13_12-02-18.cnf --user=brman --host=127.0.0.1 --port=3318 --no-timestamp --backup --target-dir=/home/mysql/bck/daily/bck_full_2018-08-13_12-02-18
180813 12:02:19 Connecting to MySQL server host: 127.0.0.1, user: brman, password: set, port: 3318, socket: not set
Using server version 10.3.7-MariaDB
/home/mysql/product/mariadb-10.3/bin/mariabackup based on MariaDB server 10.3.7-MariaDB Linux (x86_64)
mariabackup: uses posix_fadvise().
mariabackup: cd to /home/mysql/database/mariadb-103/data/
mariabackup: open files limit requested 0, set to 1024
mariabackup: using the following InnoDB configuration:
mariabackup:   innodb_data_home_dir =
mariabackup:   innodb_data_file_path = ibdata1:12M:autoextend
mariabackup:   innodb_log_group_home_dir = ./
2018-08-13 12:02:19 0 [Note] InnoDB: Number of pools: 1
mariabackup: Generating a list of tablespaces
2018-08-13 12:02:19 0 [Warning] InnoDB: Allocated tablespace ID 59 for mysql/transaction_registry, old maximum was 0
180813 12:02:19 >> log scanned up to (15975835)
180813 12:02:19 [01] Copying ibdata1 to /home/mysql/bck/daily/bck_full_2018-08-13_12-02-18/ibdata1
180813 12:02:19 [01]        ...done
...

Then brman 2.0.0 supports seamlessly all three physical backup methods (mariabackup, xtrabackup, mysqlbackup) in their newest release.

On a customer request we have added the option --pass-through to pass additional specific options through to the final back-end application (mysqldump, mariabackup, xtrabackup, mysqlbackup):

As an example the customer wanted to pass through the option --ignore-table to mysqldump:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=schema --mode=logical --policy=daily --schema=+world --pass-through="--ignore-table=world.CountryLanguage"

...
Start backup at 2018-08-13_12-11-40
  Schema to backup: world

  schema_name      engine  cnt  data_bytes    index_bytes   table_rows 
  world            InnoDB     3        655360             0        5411

  Binary logging is disabled.
  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --databases 'world' --events --ignore-table=world.CountryLanguage
  to Destination: /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  Backup size is 217054
  Backup does NOT contain any binary log information.
  Do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  md5sum --binary /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
  md5 = f07e319c36ee7bb1e662008c4c66a35a
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_schema_2018-08-13_12-11-40.sql
End backup at 2018-08-13 12:11:40 (rc=0)

In the field it is sometimes wanted to not purge the binary logs during a binlog backup. So we added the option --no-purge to not purge binary logs during binlog backup. It looked like this before:

shell> bman --target=brman:secret@127.0.0.1:3326 --type=binlog --policy=binlog
...
Start backup at 2018-08-13_12-16-48
  Binlog Index file is: /home/mysql/database/mysql-80/data/binlog.index
  Getting lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  Releasing lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  FLUSH /*!50503 BINARY */ LOGS

  Copy /home/mysql/database/mysql-80/data/binlog.000006 to /home/mysql/bck/binlog/bck_binlog.000006
  Binary log binlog.000006 begin datetime is: 2018-08-13 12:14:14 and end datetime is: 2018-08-13 12:14:30
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000006
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000006
  md5 = a7ae2a271a6c90b0bb53c562c87f6f7a
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000006
  PURGE BINARY LOGS TO 'binlog.000007'

  Copy /home/mysql/database/mysql-80/data/binlog.000007 to /home/mysql/bck/binlog/bck_binlog.000007
  Binary log binlog.000007 begin datetime is: 2018-08-13 12:14:30 and end datetime is: 2018-08-13 12:14:31
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000007
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000007
  md5 = 5b592e597241694944d70849d7a05f53
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000007
  PURGE BINARY LOGS TO 'binlog.000008'
...

and like this after:

shell> bman --target=brman:secret@127.0.0.1:3326 --type=binlog --policy=binlog --no-purge

...
Start backup at 2018-08-13_12-18-52
  Binlog Index file is: /home/mysql/database/mysql-80/data/binlog.index
  Getting lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  Releasing lock: /home/mysql/product/brman-2.0.0/lck/binlog-logical-binlog.lock
  FLUSH /*!50503 BINARY */ LOGS

  Copy /home/mysql/database/mysql-80/data/binlog.000015 to /home/mysql/bck/binlog/bck_binlog.000015
  Binary log binlog.000015 begin datetime is: 2018-08-13 12:16:48 and end datetime is: 2018-08-13 12:18:41
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000015
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000015
  md5 = 1f9a79c3ad081993b4006c58bf1d6bee
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000015

  Copy /home/mysql/database/mysql-80/data/binlog.000016 to /home/mysql/bck/binlog/bck_binlog.000016
  Binary log binlog.000016 begin datetime is: 2018-08-13 12:18:41 and end datetime is: 2018-08-13 12:18:42
  Do MD5 checksum of /home/mysql/bck/binlog/bck_binlog.000016
  md5sum --binary /home/mysql/bck/binlog/bck_binlog.000016
  md5 = ef1613e99bbfa78f75daa5ba543e3213
  /usr/bin/pigz -6 /home/mysql/bck/binlog/bck_binlog.000016
...

To make the logical backup (mysqldump) slightly faster we added the --quick option. This is done automatically and you cannot influence this behaviour.

  /home/mysql/product/mariadb-10.3/bin/mysqldump --user=brman --host=127.0.0.1 --port=3318  --quick --single-transaction --flush-logs --triggers --routines --hex-blob --events

Some of our customers use brman in combination with MyEnv and they want to have an overview of used software. So we made the version output of brman MyEnv compliant:

mysql@chef:~ [mariadb-103, 3318]> V

The following FromDual Toolbox Packages are installed:
------------------------------------------------------------------------
MyEnv:           2.0.0
BRman:           2.0.0
OpsCenter:       0.4.0
Fpmmm:           1.0.1
Nagios plug-ins: 1.0.1
O/S:             Linux / Ubuntu
Binaries:        mysql-5.7
                 mysql-8.0
                 mariadb-10.2
                 mariadb-10.3
------------------------------------------------------------------------

mysql@chef:~ [mariadb-103, 3318]>

In MySQL 5.7 general tablespaces were introduced. The utility mysqldump is not aware of general tablespaces and does not dump this information. This leads to errors during restore. FromDual brman checks for general tablespaces and writes them to the backup log so you can later extract this information at least from there. We consider this as a bug in mysqldump. MariaDB up to 10.3 has not implemented this feature yet so it is not affected of this problem.

...
Start backup at 2018-08-13_12-25-46
  WARNING: 5 general tablespaces found! mysqldump does NOT dump tablespace creation statements.

  CREATE TABLESPACE `brman_test_ts` ADD DATAFILE './brman_test_ts.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts2` ADD DATAFILE './ts2.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts3` ADD DATAFILE './ts3.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts4` ADD DATAFILE './ts4.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
  CREATE TABLESPACE `ts1` ADD DATAFILE './ts1.ibd' FILE_BLOCK_SIZE=4096 ENGINE=InnoDB
...

FromDual brman backups are quite complex and can run quite some long time thus timestamps are logged so we can find out where the time is spent or where the bottlenecks are:

...
  At 2018-08-13 12:27:17 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  md5 = d41d8cd98f00b204e9800998ecf8427e
  At 2018-08-13 12:27:17 compress file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ib_logfile0

  At 2018-08-13 12:27:18 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  md5sum --binary /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  md5 = 097ab6d70eefb6e8735837166cd4ba54
  At 2018-08-13 12:27:18 compress file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1
  /usr/bin/pigz -6 /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/ibdata1

  At 2018-08-13 12:27:19 do MD5 checksum of uncompressed file /home/mysql/bck/daily/bck_full_2018-08-13_12-27-16/xtrabackup_binlog_pos_innodb
...

A general FromDual policy is to not use the MariaDB/MySQL root user for anything except direct DBA interventions. So backup should be done with its own user. FromDual suggest brman as a username and the utility complains with a warning if root is used:

shell> bman --target=root@127.0.0.1:3318 --type=full --policy=daily
...
Start backup at 2018-08-13_12-30-29

WARNING: You should NOT use the root user for backup. Please create another user as follows:

         CREATE USER 'brman'@'127.0.0.1' IDENTIFIED BY 'S3cret123';
         GRANT ALL ON *.* TO 'brman'@'127.0.0.1';

         If you want to be more restrictive you can grant privileges as follows:

         GRANT SELECT, LOCK TABLES, RELOAD, PROCESS, TRIGGER, SUPER, REPLICATION CLIENT, SHOW VIEW, EVENT ON *.* TO 'brman'@'127.0.0.1';

         Additionally for MySQL Enterprise Backup (MEB):

         GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_progress TO 'brman'@'127.0.0.1';
         GRANT CREATE, INSERT, SELECT, DROP, UPDATE ON mysql.backup_history TO 'brman'@'127.0.0.1';
         GRANT FILE ON *.* TO 'brman'@'127.0.0.1';
         GRANT CREATE, INSERT, DROP, UPDATE ON mysql.backup_sbt_history TO 'brman'@'127.0.0.1';

         Additionally for MariaBackup / XtraBackup:

         GRANT INSERT, SELECT ON PERCONA_SCHEMA.xtrabackup_history TO 'brman'@'127.0.0.1';
...

Some customers have implemented a monitoring solution. FromDual brman can report backup return code, backup run time and backup size to the FromDual Performance Monitor for MariaDB and MySQL (fpmmm/Zabbix) now:

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --policy=daily --fpmmm-hostname=mariadb-103 --fpmmm-cache-file=/var/cache/fpmmm/fpmmm.FromDual.mariadb-103.cache
...

shell> cat /var/cache/fpmmm/fpmmm.FromDual.mariadb-103.cache
mariadb-103 FromDual.MySQL.backup.full_logical_rc 1534156619 "0"
mariadb-103 FromDual.MySQL.backup.full_logical_duration 1534156619 "129"
mariadb-103 FromDual.MySQL.backup.full_logical_size 1534156619 "7324744568"

Some customers run their databases on shared hosting systems or in cloud solutions where they do not have all the needed database privileges. For those users FromDual brman is much less intrusive now and allows backups on those restricted systems as well:

#
# /home/shinguz/etc/brman.conf
#

policy                = daily
target                = shinguz_brman:secret@localhost
type                  = schema
per-schema            = on
schema                = -shinguz_shinguz
log                   = /home/shinguz/log/bman_backup.log
backupdir             = /home/shinguz/bck


shell> /home/shinguz/brman/bin/bman --config=/home/shinguz/etc/brman.conf 1>/dev/null
...
  WARNING: Binary logging is enabled but you are lacking REPLICATION CLIENT privilege. I cannot get Master Log File and Pos!
  WARNING: I cannot check for GENERAL tablespaces. I lack the PROCESS privilege. This backup might not restore in case of presence of GENERAL tablespaces.
...

Details: Check for binary logging is made less intrusive. If RELOAD privilege is missing --master-data and/or --flush-logs options are omitted. Schema backup does not require SHOW DATABASES privilege any more.

Some customers want to push theire backups directly to an other server during backup (not pull from somewhere else). For those customers the new option --archivedestination was introduced which replaces the less powerfull option --archivedir which is deprecated. So archiving with rsync, scp and sftp is possible now (NFS mounts was possible before already):

shell> bman --target=brman:secret@127.0.0.1:3318 --type=full --policy=daily --archivedestination=sftp://oli@backup.fromdual.com:22/home/oli/bck/production/daily/
...
  /home/mysql/product/mysql-5.7.21/bin/mysqldump --user=root --host=127.0.0.1 --port=33006 --master-data=2 --quick --single-transaction --triggers --routines --hex-blob --events 'tellmatic'
  to Destination: /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  Backup size is 602021072
  Binlog file is mysql-bin.019336 and position is 287833
  Do MD5 checksum of uncompressed file /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  md5sum --binary /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  md5 = 06e1a0acd5da8acf19433b192259c1e1
  /usr/bin/pigz -6 /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql
  Archiving /home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql.gz to sftp://oli@backup.example.com:/home/oli/bck/production/daily/
  echo 'put "/home/mysql/backup/daily/bck_schema_tellmatic_2018-08-13_11-41-26.sql.gz"' | sftp -b - -oPort=22 oli@backup.fromdual.com:/home/oli/bck/production/daily/

End backup at 2018-08-13 11:42:19 (rc=0)

Schulung MariaDB/MySQL für Fortgeschrittene vom 12.-16. 11. in Köln

$
0
0

In unserer FromDual Schulung MariaDB/MySQL für Fortgeschrittene vom 12. bis 16. November in Köln hat es zur Zeit noch 3 Plätze frei.

Anmelden können Sie sich direkt bei unserem Schulungspartner GfU in Köln.

Den Schulungsinhalt finden Sie auf der FromDual Webseite.

Wenn Sie spezifische in Haus Schulungen oder eine auf Sie zugeschnittene MariaDB oder MySQL Beratung benötigen, nehmen Sie mit uns Kontakt auf.

Taxonomy upgrade extras: 

Beware of large MySQL max_sort_length parameter

$
0
0

Today we had a very interesting phenomena at a customer. He complained that MySQL always get some errors of the following type:

[ERROR] mysqld: Sort aborted: Error writing file '/tmp/MYGbBrpA' (Errcode: 28 - No space left on device)

After a first investigation we found that df -h /tmp shows from time to time a full disk but we could not see any file with ls -la /tmp/MY*.

After some more investigation we found even the query from the Slow Query Log which was producing the same problem. It looked similar to this query:

SELECT * FROM test ORDER BY field5, field4, field3, field2, field1;

Now we were capable to simulate the problem at will with the following table:

CREATE TABLE `test` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `data` varchar(64) DEFAULT NULL,
  `ts` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `field1` varchar(16) DEFAULT NULL,
  `field2` varchar(16) DEFAULT NULL,
  `field3` varchar(255) DEFAULT NULL,
  `field4` varchar(255) DEFAULT NULL,
  `field5` varchar(32) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=8912746 DEFAULT CHARSET=utf8
;

An we have seen the query in SHOW PROCESSLIST:

| Query   |   26 | Creating sort index | select * from test order by field5, field4, field3, field2, field1 |

But we were still not capable to see who or better how the hell mysqld is filling our disk!

I remembered further that I have seen some strange settings in the my.cnf before when we did the review of the database configuration. But I ignored them somehow.

[mysqld]
max_sort_length  = 8M
sort_buffer_size = 20M

Now I remembered again these settings. We changed max_sort_length back to default 1k and suddenly our space problems disappeared!

We played a bit around with different values of max_sort_length and got the following execution times for our query:

max_sort_lengthexecution time [s]comment
64 8.8 s
128 8.2 s
256 9.3 s
512 11.8 s
1k 14.9 s
2k 20.0 s
8k129.0 s
8M 75.0 sdisk full (50 G)

Conclusion

We set the values of max_sort_length back to the defaults. Our problems disappeared and we got working and much faster SELECT queries.

Do not needlessly change default values of MySQL without proving the impact. It can become worse than before!!!

The default value of max_sort_length is a good compromise between performance and an appropriate sort length.

Addendum

What I really did not like on this solution was, that I did not understand the way the problem occurred. So I did some more investigation in this. We were discussing forth and back if this could be because of XFS, because of sparse files or some kind of memory mapped files (see also man mmap).

At the end I had the idea to look at the lsof command during my running query:

mysql> SELECT * FROM test ORDER BY field5, field4, field3, field2, field1;
ERROR 3 (HY000): Error writing file '/tmp/MYBuWcXP' (Errcode: 28 - No space left on device)

shell> lsof -p 14733

COMMAND   PID  USER   FD   TYPE             DEVICE   SIZE/OFF     NODE NAME
mysqld  14733 mysql   32u   REG               8,18 9705619456 30147474 /tmp/MYck8vf4 (deleted)
mysqld  14733 mysql   49u   REG               8,18  749797376 30147596 /tmp/MYBuWcXP (deleted)

So it looks like that there were some deleted files which were growing!

Further information from the IRC channel led me to the libctemporary files (see also man 3 tmpfile).

And some hints from MadMerlin|work pointed me to:

shell> ls /proc//fd

Where you can also see those temporary files.

Thanks to MadMerlin|work for the hints!

Taxonomy upgrade extras: 

How to Install MySQL Enterprise Server 5.7 on Red Hat Enterprise Linux 7

$
0
0

If you have already Red Hat Enterprise Linux 7 (RHEL 7) or CentOS 7 in use you probably found out, that it is not that easy to install MySQL Enterprise Server because there are already pre-installed MariaDB 5.5 libraries:

shell> yum localinstall mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-libs-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-common-5.7.16-1.1.el7.x86_64.rpm
Loaded plugins: fastestmirror
Examining mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-client-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm to be installed
Examining mysql-commercial-libs-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-libs-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-libs-5.7.16-1.1.el7.x86_64.rpm to be installed
Examining mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-server-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm to be installed
Examining mysql-commercial-common-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-common-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-common-5.7.16-1.1.el7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be obsoleted
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
Loading mirror speeds from cached hostfile
 * base: wftp.tu-chemnitz.de
 * extras: wftp.tu-chemnitz.de
 * updates: mirror.rackspeed.de
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
---> Package mysql-commercial-client.x86_64 0:5.7.16-1.1.el7 will be installed
---> Package mysql-commercial-common.x86_64 0:5.7.16-1.1.el7 will be installed
---> Package mysql-commercial-libs.x86_64 0:5.7.16-1.1.el7 will be obsoleting
---> Package mysql-commercial-server.x86_64 0:5.7.16-1.1.el7 will be installed
--> Processing Dependency: /usr/bin/perl for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: net-tools for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(Getopt::Long) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(strict) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1()(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Running transaction check
---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be obsoleted
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
---> Package net-tools.x86_64 0:2.0-0.17.20131004git.el7 will be installed
---> Package perl.x86_64 4:5.16.3-286.el7 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-286.el7 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Unix) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Functions) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Cwd) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-286.el7.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-2.el7 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be obsoleted
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
---> Package perl-Carp.noarch 0:1.26-244.el7 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.el7 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.el7 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.el7 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.el7 will be installed
---> Package perl-PathTools.x86_64 0:3.40-5.el7 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.el7 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.el7 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.el7.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.el7.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 will be installed
---> Package perl-Socket.x86_64 0:2.010-3.el7 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.el7 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.el7 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.el7 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.el7 will be installed
---> Package perl-constant.noarch 0:1.27-2.el7 will be installed
---> Package perl-libs.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-macros.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-threads.x86_64 0:1.87-4.el7 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.el7 will be installed
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be obsoleted
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
---> Package perl-Encode.x86_64 0:2.51-7.el7 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-286.el7 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.el7 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.el7 will be installed
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be obsoleted
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: 2:postfix-2.10.1-6.el7.x86_64
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.el7 will be installed
---> Package perl-parent.noarch 1:0.225-244.el7 will be installed
--> Finished Dependency Resolution
Error: Package: 2:postfix-2.10.1-6.el7.x86_64 (@anaconda)
           Requires: libmysqlclient.so.18(libmysqlclient_18)(64bit)
           Removing: 1:mariadb-libs-5.5.50-1.el7_2.x86_64 (@updates)
               libmysqlclient.so.18(libmysqlclient_18)(64bit)
           Obsoleted By: mysql-commercial-libs-5.7.16-1.1.el7.x86_64 (/mysql-commercial-libs-5.7.16-1.1.el7.x86_64)
               Not found
           Available: 1:mariadb-libs-5.5.44-2.el7.centos.x86_64 (base)
               libmysqlclient.so.18(libmysqlclient_18)(64bit)
           Available: 1:mariadb-libs-5.5.47-1.el7_2.x86_64 (updates)
               libmysqlclient.so.18(libmysqlclient_18)(64bit)
Error: Package: 2:postfix-2.10.1-6.el7.x86_64 (@anaconda)
           Requires: libmysqlclient.so.18()(64bit)
           Removing: 1:mariadb-libs-5.5.50-1.el7_2.x86_64 (@updates)
               libmysqlclient.so.18()(64bit)
           Obsoleted By: mysql-commercial-libs-5.7.16-1.1.el7.x86_64 (/mysql-commercial-libs-5.7.16-1.1.el7.x86_64)
              ~libmysqlclient.so.20()(64bit)
           Available: 1:mariadb-libs-5.5.44-2.el7.centos.x86_64 (base)
               libmysqlclient.so.18()(64bit)
           Available: 1:mariadb-libs-5.5.47-1.el7_2.x86_64 (updates)
               libmysqlclient.so.18()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

You can remove postfix and mariadb-libs but this will most probably cause you troubles sooner or later:

shell> yum remove postfix mariadb-libs
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.50-1.el7_2 will be erased
---> Package postfix.x86_64 2:2.10.1-6.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================
 Package                                       Arch                                    Version                                           Repository                                  Size
==========================================================================================================================================================================================
Removing:
 mariadb-libs                                  x86_64                                  1:5.5.50-1.el7_2                                  @updates                                   4.4 M
 postfix                                       x86_64                                  2:2.10.1-6.el7                                    @anaconda                                   12 M

Transaction Summary
==========================================================================================================================================================================================
Remove  2 Packages

Installed size: 17 M
Is this ok [y/N]: n

If you look at the MySQL Installation Guide: Installing MySQL on Linux Using RPM Packages from Oracle it is unfortunately not really very well documented how to install MySQL Enterprise Server: "using rpm -Uvh instead makes the installation process more prone to failure, due to potential dependency issues the installation process might run into".

So the correct way to install MySQL Enterprise Server 5.7 on Red Hat Enterprise Linux 7 (RHEL7) or CentOS 7 is as follows:

shell> rpm --upgrade -vh mysql-commercial-libs-compat-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-libs-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-common-5.7.16-1.1.el7.x86_64.rpmshell> yum localinstall mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm
Loaded plugins: fastestmirror
Examining mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-server-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm to be installed
Examining mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm: mysql-commercial-client-5.7.16-1.1.el7.x86_64
Marking mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package mysql-commercial-client.x86_64 0:5.7.16-1.1.el7 will be installed
---> Package mysql-commercial-server.x86_64 0:5.7.16-1.1.el7 will be installed
--> Processing Dependency: /usr/bin/perl for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
Loading mirror speeds from cached hostfile
 * base: wftp.tu-chemnitz.de
 * extras: wftp.tu-chemnitz.de
 * updates: mirror.rackspeed.de
--> Processing Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: net-tools for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(Getopt::Long) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(strict) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1()(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Running transaction check
---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed
---> Package net-tools.x86_64 0:2.0-0.17.20131004git.el7 will be installed
---> Package perl.x86_64 4:5.16.3-286.el7 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-286.el7 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Unix) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Functions) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Cwd) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-286.el7.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-2.el7 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Running transaction check
---> Package perl-Carp.noarch 0:1.26-244.el7 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.el7 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.el7 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.el7 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.el7 will be installed
---> Package perl-PathTools.x86_64 0:3.40-5.el7 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.el7 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.el7 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.el7.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.el7.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 will be installed
---> Package perl-Socket.x86_64 0:2.010-3.el7 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.el7 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.el7 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.el7 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.el7 will be installed
---> Package perl-constant.noarch 0:1.27-2.el7 will be installed
---> Package perl-libs.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-macros.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-threads.x86_64 0:1.87-4.el7 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.el7 will be installed
--> Running transaction check
---> Package perl-Encode.x86_64 0:2.51-7.el7 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-286.el7 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.el7 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.el7 will be installed
--> Running transaction check
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.el7 will be installed
---> Package perl-parent.noarch 1:0.225-244.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================
 Package                                    Arch                      Version                                     Repository                                                         Size
==========================================================================================================================================================================================
Installing:
 mysql-commercial-client                    x86_64                    5.7.16-1.1.el7                              /mysql-commercial-client-5.7.16-1.1.el7.x86_64                     94 M
 mysql-commercial-server                    x86_64                    5.7.16-1.1.el7                              /mysql-commercial-server-5.7.16-1.1.el7.x86_64                    721 M
Installing for dependencies:
 libaio                                     x86_64                    0.3.109-13.el7                              base                                                               24 k
 net-tools                                  x86_64                    2.0-0.17.20131004git.el7                    base                                                              304 k
 perl                                       x86_64                    4:5.16.3-286.el7                            base                                                              8.0 M
 perl-Carp                                  noarch                    1.26-244.el7                                base                                                               19 k
 perl-Encode                                x86_64                    2.51-7.el7                                  base                                                              1.5 M
 perl-Exporter                              noarch                    5.68-3.el7                                  base                                                               28 k
 perl-File-Path                             noarch                    2.09-2.el7                                  base                                                               26 k
 perl-File-Temp                             noarch                    0.23.01-3.el7                               base                                                               56 k
 perl-Filter                                x86_64                    1.49-3.el7                                  base                                                               76 k
 perl-Getopt-Long                           noarch                    2.40-2.el7                                  base                                                               56 k
 perl-HTTP-Tiny                             noarch                    0.033-3.el7                                 base                                                               38 k
 perl-PathTools                             x86_64                    3.40-5.el7                                  base                                                               82 k
 perl-Pod-Escapes                           noarch                    1:1.04-286.el7                              base                                                               50 k
 perl-Pod-Perldoc                           noarch                    3.20-4.el7                                  base                                                               87 k
 perl-Pod-Simple                            noarch                    1:3.28-4.el7                                base                                                              216 k
 perl-Pod-Usage                             noarch                    1.63-3.el7                                  base                                                               27 k
 perl-Scalar-List-Utils                     x86_64                    1.27-248.el7                                base                                                               36 k
 perl-Socket                                x86_64                    2.010-3.el7                                 base                                                               49 k
 perl-Storable                              x86_64                    2.45-3.el7                                  base                                                               77 k
 perl-Text-ParseWords                       noarch                    3.29-4.el7                                  base                                                               14 k
 perl-Time-HiRes                            x86_64                    4:1.9725-3.el7                              base                                                               45 k
 perl-Time-Local                            noarch                    1.2300-2.el7                                base                                                               24 k
 perl-constant                              noarch                    1.27-2.el7                                  base                                                               19 k
 perl-libs                                  x86_64                    4:5.16.3-286.el7                            base                                                              687 k
 perl-macros                                x86_64                    4:5.16.3-286.el7                            base                                                               43 k
 perl-parent                                noarch                    1:0.225-244.el7                             base                                                               12 k
 perl-podlators                             noarch                    2.5.1-3.el7                                 base                                                              112 k
 perl-threads                               x86_64                    1.87-4.el7                                  base                                                               49 k
 perl-threads-shared                        x86_64                    1.43-6.el7                                  base                                                               39 k

Transaction Summary
==========================================================================================================================================================================================
Install  2 Packages (+29 Dependent packages)

Total size: 826 M
Total download size: 12 M
Installed size: 852 M
Is this ok [y/d/N]: y

Further information you can find in the slides of my colleague Jörg Brühe: MySQL selection also against the distirbution he presented at FrOSCon 2016.

How to build your own RPM repository for MySQL Enterprise Server 5.7 on RHEL 7

$
0
0

Prepare the RPM repository server

Install the software to create a RPM repository server (on an Ubuntu web server):

shell> sudo apt-get install createrepo

Create the directory structures and initialize the repositories:

shell> sudo mkdir -p /var/www/html/repos/centos/7/os/x86_64
shell> sudo createrepo /var/www/html/repos/centos/7/os/x86_64/

Copy the MySQL Enterprise RPM packages to the repository:

shell> sudo cd /var/www/html/repos/centos/7/os/x86_64/
shell> sudo unzip -v /download/V790254-01.zip   # MySQL Enterprise Server 5.7.16
Archive:  /download/V790254-01.zip
   Length  Method      Size Cmpr    Date    Time   CRC-32   Name
---------  ------  -------- ---- ---------- ----- --------  ----
 23979568  Stored  23979568   0% 2016-09-29 16:56 ae6693b1  mysql-commercial-client-5.7.16-1.1.el7.x86_64.rpm
 45327096  Stored  45327096   0% 2016-09-29 16:56 bb83c965  mysql-commercial-embedded-5.7.16-1.1.el7.x86_64.rpm
   271300  Stored    271300   0% 2016-09-29 16:56 b4eebc96  mysql-commercial-common-5.7.16-1.1.el7.x86_64.rpm
  2041404  Stored   2041404   0% 2016-09-29 16:57 cfc63d84  mysql-commercial-libs-compat-5.7.16-1.1.el7.x86_64.rpm
 23747244  Stored  23747244   0% 2016-09-29 16:56 7b4d36fa  mysql-commercial-embedded-compat-5.7.16-1.1.el7.x86_64.rpm
  3644756  Stored   3644756   0% 2016-09-29 16:56 613af6a3  mysql-commercial-devel-5.7.16-1.1.el7.x86_64.rpm
170697184  Stored 170697184   0% 2016-09-29 16:57 ce26e84a  mysql-commercial-server-5.7.16-1.1.el7.x86_64.rpm
116778644  Stored 116778644   0% 2016-09-29 16:58 8ca7f0f0  mysql-commercial-test-5.7.16-1.1.el7.x86_64.rpm
125679224  Stored 125679224   0% 2016-09-29 16:57 e74a416d  mysql-commercial-embedded-devel-5.7.16-1.1.el7.x86_64.rpm
  2172140  Stored   2172140   0% 2016-09-29 16:57 e2a1c2f9  mysql-commercial-libs-5.7.16-1.1.el7.x86_64.rpm
     3018  Stored      3018   0% 2016-10-11 13:00 6d6efc1e  README.txt
---------         --------- ----                            -------
514341578         514341578   0%                            11 files

Update the RPM repository meta data:

shell> sudo createrepo --update /var/www/html/repos/centos/7/os/x86_64/

Now the RPM repository server is finished.

Prepare the RPM client

Create the MySQL Enterprise Server repository file for the client. The IP address represents the IP address of your RPM repository server:

#
# /etc/yum.repos.d/mysql-enterprise.repo
#

[mysql57-enterprise]

name=MySQL 5.7 Enterprise Server
baseurl=http://192.168.56.1/repos/centos/$releasever/os/$basearch/
gpgcheck=1
enabled=1
gpgkey=http://192.168.56.1/repos/centos/RPM-GPG-KEY-CentOS-7

Clean up the yum cache:

shell> yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base extras mysql57-enterprise updates
Cleaning up everything

Show all know RPM repositories:

shell> yum repolist
Loaded plugins: fastestmirror
base                                                                                                                                                               | 3.6 kB  00:00:00     
extras                                                                                                                                                             | 3.4 kB  00:00:00     
mysql57-enterprise                                                                                                                                                 | 2.9 kB  00:00:00     
updates                                                                                                                                                            | 3.4 kB  00:00:00     
(1/5): mysql57-enterprise/7/x86_64/primary_db                                                                                                                      |  12 kB  00:00:00     
(2/5): base/7/x86_64/group_gz                                                                                                                                      | 155 kB  00:00:00     
(3/5): extras/7/x86_64/primary_db                                                                                                                                  | 166 kB  00:00:00     
(4/5): base/7/x86_64/primary_db                                                                                                                                    | 5.3 MB  00:00:06     
(5/5): updates/7/x86_64/primary_db                                                                                                                                 | 8.4 MB  00:00:11     
Determining fastest mirrors
 * base: mirror.imt-systems.com
 * extras: mirror.eu.oneandone.net
 * updates: mirror.rackspeed.de
repo id                                                                                   repo name                                                                                 status
base/7/x86_64                                                                             CentOS-7 - Base                                                                           9'007
extras/7/x86_64                                                                           CentOS-7 - Extras                                                                           393
mysql57-enterprise/7/x86_64                                                               MySQL 5.7 Enterprise Server                                                                  10
updates/7/x86_64                                                                          CentOS-7 - Updates                                                                        2'548
repolist: 11'958

Search what MySQL Enterprise Server packages are available:

shell> yum search mysql-commercial
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.imt-systems.com
 * extras: mirror.eu.oneandone.net
 * updates: mirror.rackspeed.de
============================================================================= N/S matched: mysql-commercial ==============================================================================
mysql-commercial-client.x86_64 : MySQL database client applications and tools
mysql-commercial-common.x86_64 : MySQL database common files for server and client libs
mysql-commercial-devel.x86_64 : Development header files and libraries for MySQL database client applications
mysql-commercial-embedded.x86_64 : MySQL embedded library
mysql-commercial-embedded-compat.x86_64 : MySQL embedded compat library
mysql-commercial-embedded-devel.x86_64 : Development header files and libraries for MySQL as an embeddable library
mysql-commercial-libs.x86_64 : Shared libraries for MySQL database client applications
mysql-commercial-libs-compat.x86_64 : Shared compat libraries for MySQL 5.6.31 database client applications
mysql-commercial-server.x86_64 : A very fast and reliable SQL database server
mysql-commercial-test.x86_64 : Test suite for the MySQL database server

  Name and summary matches only, use "search all" for everything.

Now you are ready to install your MySQL Enterprise Server RPM packages:

shell> yum install mysql-commercial-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.imt-systems.com
 * extras: mirror.eu.oneandone.net
 * updates: mirror.rackspeed.de
Resolving Dependencies
--> Running transaction check
---> Package mysql-commercial-server.x86_64 0:5.7.16-1.1.el7 will be installed
--> Processing Dependency: mysql-commercial-client(x86-64) >= 5.7.9 for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(Getopt::Long) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: /usr/bin/perl for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: perl(strict) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: net-tools for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Processing Dependency: libaio.so.1()(64bit) for package: mysql-commercial-server-5.7.16-1.1.el7.x86_64
--> Running transaction check
---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed
---> Package mysql-commercial-client.x86_64 0:5.7.16-1.1.el7 will be installed
---> Package net-tools.x86_64 0:2.0-0.17.20131004git.el7 will be installed
---> Package perl.x86_64 4:5.16.3-286.el7 will be installed
--> Processing Dependency: perl-libs = 4:5.16.3-286.el7 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) >= 1.3 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) >= 1.10 for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-macros for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl-libs for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads::shared) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(threads) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(constant) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::Local) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Time::HiRes) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Storable) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Socket) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Scalar::Util) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::XHTML) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Pod::Simple::Search) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Temp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Unix) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec::Functions) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Spec) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(File::Path) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Exporter) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Cwd) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: perl(Carp) for package: 4:perl-5.16.3-286.el7.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-286.el7.x86_64
---> Package perl-Getopt-Long.noarch 0:2.40-2.el7 will be installed
--> Processing Dependency: perl(Pod::Usage) >= 1.14 for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Processing Dependency: perl(Text::ParseWords) for package: perl-Getopt-Long-2.40-2.el7.noarch
--> Running transaction check
---> Package perl-Carp.noarch 0:1.26-244.el7 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.el7 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.el7 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.el7 will be installed
---> Package perl-Filter.x86_64 0:1.49-3.el7 will be installed
---> Package perl-PathTools.x86_64 0:3.40-5.el7 will be installed
---> Package perl-Pod-Simple.noarch 1:3.28-4.el7 will be installed
--> Processing Dependency: perl(Pod::Escapes) >= 1.04 for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
--> Processing Dependency: perl(Encode) for package: 1:perl-Pod-Simple-3.28-4.el7.noarch
---> Package perl-Pod-Usage.noarch 0:1.63-3.el7 will be installed
--> Processing Dependency: perl(Pod::Text) >= 3.15 for package: perl-Pod-Usage-1.63-3.el7.noarch
--> Processing Dependency: perl-Pod-Perldoc for package: perl-Pod-Usage-1.63-3.el7.noarch
---> Package perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 will be installed
---> Package perl-Socket.x86_64 0:2.010-3.el7 will be installed
---> Package perl-Storable.x86_64 0:2.45-3.el7 will be installed
---> Package perl-Text-ParseWords.noarch 0:3.29-4.el7 will be installed
---> Package perl-Time-HiRes.x86_64 4:1.9725-3.el7 will be installed
---> Package perl-Time-Local.noarch 0:1.2300-2.el7 will be installed
---> Package perl-constant.noarch 0:1.27-2.el7 will be installed
---> Package perl-libs.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-macros.x86_64 4:5.16.3-286.el7 will be installed
---> Package perl-threads.x86_64 0:1.87-4.el7 will be installed
---> Package perl-threads-shared.x86_64 0:1.43-6.el7 will be installed
--> Running transaction check
---> Package perl-Encode.x86_64 0:2.51-7.el7 will be installed
---> Package perl-Pod-Escapes.noarch 1:1.04-286.el7 will be installed
---> Package perl-Pod-Perldoc.noarch 0:3.20-4.el7 will be installed
--> Processing Dependency: perl(parent) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
--> Processing Dependency: perl(HTTP::Tiny) for package: perl-Pod-Perldoc-3.20-4.el7.noarch
---> Package perl-podlators.noarch 0:2.5.1-3.el7 will be installed
--> Running transaction check
---> Package perl-HTTP-Tiny.noarch 0:0.033-3.el7 will be installed
---> Package perl-parent.noarch 1:0.225-244.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================
 Package                                           Arch                             Version                                            Repository                                    Size
==========================================================================================================================================================================================
Installing:
 mysql-commercial-server                           x86_64                           5.7.16-1.1.el7                                     mysql57-enterprise                           163 M
Installing for dependencies:
 libaio                                            x86_64                           0.3.109-13.el7                                     base                                          24 k
 mysql-commercial-client                           x86_64                           5.7.16-1.1.el7                                     mysql57-enterprise                            23 M
 net-tools                                         x86_64                           2.0-0.17.20131004git.el7                           base                                         304 k
 perl                                              x86_64                           4:5.16.3-286.el7                                   base                                         8.0 M
 perl-Carp                                         noarch                           1.26-244.el7                                       base                                          19 k
 perl-Encode                                       x86_64                           2.51-7.el7                                         base                                         1.5 M
 perl-Exporter                                     noarch                           5.68-3.el7                                         base                                          28 k
 perl-File-Path                                    noarch                           2.09-2.el7                                         base                                          26 k
 perl-File-Temp                                    noarch                           0.23.01-3.el7                                      base                                          56 k
 perl-Filter                                       x86_64                           1.49-3.el7                                         base                                          76 k
 perl-Getopt-Long                                  noarch                           2.40-2.el7                                         base                                          56 k
 perl-HTTP-Tiny                                    noarch                           0.033-3.el7                                        base                                          38 k
 perl-PathTools                                    x86_64                           3.40-5.el7                                         base                                          82 k
 perl-Pod-Escapes                                  noarch                           1:1.04-286.el7                                     base                                          50 k
 perl-Pod-Perldoc                                  noarch                           3.20-4.el7                                         base                                          87 k
 perl-Pod-Simple                                   noarch                           1:3.28-4.el7                                       base                                         216 k
 perl-Pod-Usage                                    noarch                           1.63-3.el7                                         base                                          27 k
 perl-Scalar-List-Utils                            x86_64                           1.27-248.el7                                       base                                          36 k
 perl-Socket                                       x86_64                           2.010-3.el7                                        base                                          49 k
 perl-Storable                                     x86_64                           2.45-3.el7                                         base                                          77 k
 perl-Text-ParseWords                              noarch                           3.29-4.el7                                         base                                          14 k
 perl-Time-HiRes                                   x86_64                           4:1.9725-3.el7                                     base                                          45 k
 perl-Time-Local                                   noarch                           1.2300-2.el7                                       base                                          24 k
 perl-constant                                     noarch                           1.27-2.el7                                         base                                          19 k
 perl-libs                                         x86_64                           4:5.16.3-286.el7                                   base                                         687 k
 perl-macros                                       x86_64                           4:5.16.3-286.el7                                   base                                          43 k
 perl-parent                                       noarch                           1:0.225-244.el7                                    base                                          12 k
 perl-podlators                                    noarch                           2.5.1-3.el7                                        base                                         112 k
 perl-threads                                      x86_64                           1.87-4.el7                                         base                                          49 k
 perl-threads-shared                               x86_64                           1.43-6.el7                                         base                                          39 k

Transaction Summary
==========================================================================================================================================================================================
Install  1 Package (+30 Dependent packages)

Total download size: 197 M
Installed size: 852 M
Is this ok [y/d/N]: y   

Further sources of information:

What are the differences between MySQL Community and MySQL Enterprise Server 5.7

$
0
0

The MySQL Server itself

The differences between the MySQL Community Server and the MySQL Enterprise Server 5.7 are as follows as claimed by Oracle:

  • The license of the MySQL Server itself.
  • Only MySQL Enterprise Edition has the Enterprise plug-ins (Thread Pool, PAM, Audit, etc.)
  • Certifications and Indemnification support for the MySQL Enterprise Server.
  • The MySQL Community Server statically links against yaSSLand readline vs MySQL Enterprise Server against OpenSSLand libedit. This restriction seems to be lifted in MySQL 8.0.

The license of the MySQL Server

The MySQL Community Server is licensed under the GNU General Public License version 2 whereas the MySQL Enterprise Server is under an Oracle proprietary license as you can see from the following diffs of 2 random files:

shell> diff mysql-5.7.16-linux-glibc2.5-x86_64/share/charsets/latin1.xml mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share/charsets/latin1.xml
6,7c6,7
<  Copyright (c) 2003, 2005 MySQL AB
<    Use is subject to license terms
---
>    Copyright (c) 2003, 2005 MySQL AB
>    Use is subject to license terms.
9,20c9,20
<   This program is free software; you can redistribute it and/or modify
<   it under the terms of the GNU General Public License as published by
<   the Free Software Foundation; version 2 of the License.
< 
<   This program is distributed in the hope that it will be useful,
<   but WITHOUT ANY WARRANTY; without even the implied warranty of
<   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
<   GNU General Public License for more details.
< 
<   You should have received a copy of the GNU General Public License
<   along with this program; if not, write to the Free Software
<   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
---
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>    The lines above are intentionally left blank

This information can also be found in the following files:

mysql-5.7.16-linux-glibc2.5-x86_64/COPYING

                    GNU GENERAL PUBLIC LICENSE
                       Version 2, June 1991

 Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.

...


mysql-5.7.16-linux-glibc2.5-x86_64/README

MySQL Server 5.7

This is a release of MySQL, a dual-license SQL database server.
For the avoidance of doubt, this particular copy of the software
is released under the version 2 of the GNU General Public License.
MySQL is brought to you by Oracle.

...


mysql-advanced-5.7.16-linux-glibc2.5-x86_64/LICENSE.mysql

MySQL Server

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

This software and related documentation are provided under a license
agreement containing restrictions on use and disclosure and are protected by
intellectual property laws. Except as expressly permitted in your license
agreement or allowed by law, you may not use, copy, reproduce, translate,
broadcast, modify, license, transmit, distribute, exhibit, perform, publish,
or display any part, in any form, or by any means. Reverse engineering,
disassembly, or decompilation of this software, unless required by law for
interoperability, is prohibited.

...


mysql-advanced-5.7.16-linux-glibc2.5-x86_64/README

MySQL Server 5.x

This is a release of MySQL, a dual-license SQL database server.
For the avoidance of doubt, this particular copy of the software
is released under a commercial license and the GNU General Public
License does not apply. MySQL is brought to you by Oracle.

...

Enterprise plug-ins of the MySQL Enterprise Server

Oracle/MySQL follows the open core business model with the MySQL Server. This means: The MySQL Community Server is the same as the MySQL Enterprise Server but the MySQL Enterprise Server has some additional modules and programs compared to the MySQL Community Server:

  • MySQL Enterprise Backup
  • MySQL Enterprise Monitor
  • MySQL Enterprise Security
  • MySQL Enterprise Audit

See also: MySQL Enterprise Edition.

If we check this in the packages we find the following additional plug-ins in the MySQL Enterprise Server:

shell> ls -la mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin:
-rwxr-xr-x 1 mysql mysql  3556085 Sep 28 19:35 audit_log.so
-rwxr-xr-x 1 mysql mysql    73855 Sep 28 19:35 authentication_pam.so
-rwxr-xr-x 1 mysql mysql  1595720 Sep 28 19:35 firewall.so
-rwxr-xr-x 1 mysql mysql  3748543 Sep 28 19:35 keyring_okv.so
-rwxr-xr-x 1 mysql mysql  2283844 Sep 28 19:35 openssl_udf.so
-rwxr-xr-x 1 mysql mysql   567032 Sep 28 19:34 thread_pool.so

MySQL Enterprise Server Certification and Indemnification support

This is legal stuff. I only care about technical problems... If you have Open Source legal questions please get in contact with us and we will direct you to lawyers which are specialised in the Open Source field.

Different libraries

The MySQL Community Server statically links against yaSSLand readline vs MySQL Enterprise Server against OpenSSLand libedit.

This is on one side a legal problem GPL vs BSD license. On the other side it is a political problem. Unfortunately the OpenSSL used in the MySQL Enterprise Server is actually a bit more feature reach than yaSSL.

We can also see the differences between the different SSL libraries when we search for the symbols:

shell> grep -ic yassl *.mysqld
community.mysqld:1118
enterprise.mysqld:0

shell> grep -ic openssl *.mysqld
community.mysqld:3
enterprise.mysqld:38

shell]> grep -i openssl community.mysql
yaOpenSSL_add_all_algorithms
_ZL16Sys_have_openssl
_ZN8TaoCrypt18RSA_Public_Decoder17ReadHeaderOpenSSLEv

Other technical comparisons

Beside of the described findings above I am a very curious child...

It seems like we can find here some build info:

shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN    
-rw-r--r-- 1 mysql mysql 5672 Sep 28 20:14 mysql-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN
-rw-r--r-- 1 mysql mysql 5969 Sep 28 19:45 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN

shell> diff mysql-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs/INFO_BIN    
10,13c10,13
< C_FLAGS =  -fPIC -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement -O3 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -I/export/home/somepath/release/include -I/export/home/somepath/mysql-5.7.16/extra/rapidjson/include -I/export/home/somepath/release/libbinlogevents/include -I/export/home/somepath/mysql-5.7.16/libbinlogevents/export -I/export/home/somepath/mysql-5.7.16/include -I/export/home/somepath/mysql-5.7.16/sql/conn_handler -I/export/home/somepath/mysql-5.7.16/libbinlogevents/include -I/export/home/somepath/mysql-5.7.16/sql -I/export/home/somepath/mysql-5.7.16/sql/auth -I/export/home/somepath/mysql-5.7.16/regex -I/export/home/somepath/mysql-5.7.16/zlib -I/export/home/somepath/mysql-5.7.16/extra/yassl/include -I/export/home/somepath/mysql-5.7.16/extra/yassl/taocrypt/include -I/export/home/somepath/release/sql -I/export/home/somepath/mysql-5.7.16/extra/lz4    -DHAVE_YASSL -DYASSL_PREFIX -DHAVE_OPENSSL -DMULTI_THREADED< C_DEFINES = -DHAVE_CONFIG_H -DHAVE_LIBEVENT1 -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
< CXX_FLAGS =  -fPIC -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter -O3 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -I/export/home/somepath/release/include -I/export/home/somepath/mysql-5.7.16/extra/rapidjson/include -I/export/home/somepath/release/libbinlogevents/include -I/export/home/somepath/mysql-5.7.16/libbinlogevents/export -I/export/home/somepath/mysql-5.7.16/include -I/export/home/somepath/mysql-5.7.16/sql/conn_handler -I/export/home/somepath/mysql-5.7.16/libbinlogevents/include -I/export/home/somepath/mysql-5.7.16/sql -I/export/home/somepath/mysql-5.7.16/sql/auth -I/export/home/somepath/mysql-5.7.16/regex -I/export/home/somepath/mysql-5.7.16/zlib -I/export/home/somepath/mysql-5.7.16/extra/yassl/include -I/export/home/somepath/mysql-5.7.16/extra/yassl/taocrypt/include -I/export/home/somepath/release/sql -I/export/home/somepath/mysql-5.7.16/extra/lz4    -DHAVE_YASSL -DYASSL_PREFIX -DHAVE_OPENSSL -DMULTI_THREADED< CXX_DEFINES = -DHAVE_CONFIG_H -DHAVE_LIBEVENT1 -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
---
> C_FLAGS =  -fPIC -Wall -Wextra -Wformat-security -Wvla -Wwrite-strings -Wdeclaration-after-statement -O3 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -I/export/home/somepath/release/include -I/export/home/somepath/mysqlcom-pro-5.7.16/extra/rapidjson/include -I/export/home/somepath/release/libbinlogevents/include -I/export/home/somepath/mysqlcom-pro-5.7.16/libbinlogevents/export -I/export/home/somepath/mysqlcom-pro-5.7.16/include -I/export/home/somepath/mysqlcom-pro-5.7.16/sql/conn_handler -I/export/home/somepath/mysqlcom-pro-5.7.16/libbinlogevents/include -I/export/home/somepath/mysqlcom-pro-5.7.16/sql -I/export/home/somepath/mysqlcom-pro-5.7.16/sql/auth -I/export/home/somepath/mysqlcom-pro-5.7.16/regex -I/export/home/somepath/mysqlcom-pro-5.7.16/zlib -I/export/home/somepath/dep4/include -I/export/home/somepath/release/sql -I/export/home/somepath/mysqlcom-pro-5.7.16/extra/lz4   
> C_DEFINES = -DHAVE_CONFIG_H -DHAVE_LIBEVENT1 -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
> CXX_FLAGS =  -fPIC -Wall -Wextra -Wformat-security -Wvla -Woverloaded-virtual -Wno-unused-parameter -O3 -g -fabi-version=2 -fno-omit-frame-pointer -fno-strict-aliasing -DDBUG_OFF -I/export/home/somepath/release/include -I/export/home/somepath/mysqlcom-pro-5.7.16/extra/rapidjson/include -I/export/home/somepath/release/libbinlogevents/include -I/export/home/somepath/mysqlcom-pro-5.7.16/libbinlogevents/export -I/export/home/somepath/mysqlcom-pro-5.7.16/include -I/export/home/somepath/mysqlcom-pro-5.7.16/sql/conn_handler -I/export/home/somepath/mysqlcom-pro-5.7.16/libbinlogevents/include -I/export/home/somepath/mysqlcom-pro-5.7.16/sql -I/export/home/somepath/mysqlcom-pro-5.7.16/sql/auth -I/export/home/somepath/mysqlcom-pro-5.7.16/regex -I/export/home/somepath/mysqlcom-pro-5.7.16/zlib -I/export/home/somepath/dep4/include -I/export/home/somepath/release/sql -I/export/home/somepath/mysqlcom-pro-5.7.16/extra/lz4   
> CXX_DEFINES = -DHAVE_CONFIG_H -DHAVE_LIBEVENT1 -DHAVE_OPENSSL -DHAVE_REPLICATION -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE
23a24
> CRYPTO_LIBRARY:FILEPATH=/export/home/somepath/dep4/lib/libcrypto.a
34c35
< FEATURE_SET:STRING=community
---
> FEATURE_SET:STRING=xlarge
44a46,48
> OPENSSL_INCLUDE_DIR:PATH=/export/home/somepath/dep4/include
> OPENSSL_LIBRARY:FILEPATH=/export/home/somepath/dep4/lib/libssl.a
> OPENSSL_ROOT_DIR:PATH=/export/home/somepath/dep4
73c77,78
< WITH_SSL:STRING=bundled
---
> WITH_SSL:STRING=/export/home/somepath/dep4
> WITH_SSL_PATH:PATH=/export/home/somepath/dep4

The size of the contents is more or less the same (22 Mbyte difference):

shell> du -ksc mysql*5.7.16-linux*/*

1355480 mysql-5.7.16-linux-glibc2.5-x86_64/bin
20      mysql-5.7.16-linux-glibc2.5-x86_64/COPYING
20      mysql-5.7.16-linux-glibc2.5-x86_64/docs
1220    mysql-5.7.16-linux-glibc2.5-x86_64/include
1217880 mysql-5.7.16-linux-glibc2.5-x86_64/lib
836     mysql-5.7.16-linux-glibc2.5-x86_64/man
4       mysql-5.7.16-linux-glibc2.5-x86_64/README
4144    mysql-5.7.16-linux-glibc2.5-x86_64/share
32      mysql-5.7.16-linux-glibc2.5-x86_64/support-files
2579636 mysql-5.7.16-linux-glibc2.5-x86_64

1345864 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin
4       mysql-advanced-5.7.16-linux-glibc2.5-x86_64/LICENSE.mysql
15664   mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs
1188    mysql-advanced-5.7.16-linux-glibc2.5-x86_64/include
1233168 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib
988     mysql-advanced-5.7.16-linux-glibc2.5-x86_64/man
4       mysql-advanced-5.7.16-linux-glibc2.5-x86_64/README
4144    mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share
32      mysql-advanced-5.7.16-linux-glibc2.5-x86_64/support-files
2601060 mysql-advanced-5.7.16-linux-glibc2.5-x86_64

The biggest difference we see in the docs folder where the mysql.info file is located (left over from clean-up?):

shell> ll mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs
-rw-r--r-- 1 mysql mysql     1789 Sep 28 19:06 ChangeLog
-rw-r--r-- 1 mysql mysql     5969 Sep 28 19:45 INFO_BIN
-rw-r--r-- 1 mysql mysql      185 Sep 28 19:34 INFO_SRC
-rw-r--r-- 1 mysql mysql 16017476 Sep 28 19:06 mysql.info

If we want to see the different number of files:

for i in $(find mysql*5.7.16-linux* -maxdepth 1 -type d) ; do 
  echo -n $i": " ; 
  ( find $i -type f | wc -l ) ; 
done

mysql-5.7.16-linux-glibc2.5-x86_64/support-files: 5
mysql-5.7.16-linux-glibc2.5-x86_64/share: 62
mysql-5.7.16-linux-glibc2.5-x86_64/man: 41
mysql-5.7.16-linux-glibc2.5-x86_64/bin: 38
mysql-5.7.16-linux-glibc2.5-x86_64/lib: 116
mysql-5.7.16-linux-glibc2.5-x86_64/docs: 3
mysql-5.7.16-linux-glibc2.5-x86_64/include: 107
mysql-5.7.16-linux-glibc2.5-x86_64: 374

mysql-advanced-5.7.16-linux-glibc2.5-x86_64/support-files: 5
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share: 66
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/man: 41
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin: 38
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib: 128
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs: 4
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/include: 107
mysql-advanced-5.7.16-linux-glibc2.5-x86_64: 391

Those are the important differences:

-rw-r--r-- 1 mysql mysql     1046 Sep 28 19:02 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share/audit_log_filter_linux_install.sql
-rw-r--r-- 1 mysql mysql     1052 Sep 28 19:02 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share/audit_log_filter_win_install.sql
-rw-r--r-- 1 mysql mysql      239 Sep 28 19:33 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share/uninstall_rewriter.sql
-rw-r--r-- 1 mysql mysql     2207 Sep 28 19:02 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/share/win_install_firewall.sql

-rw-r--r-- 1 mysql mysql 16017476 Sep 28 19:06 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/docs/mysql.info

-rwxr-xr-x 1 mysql mysql  3556085 Sep 28 19:35 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/audit_log.so
-rwxr-xr-x 1 mysql mysql    73855 Sep 28 19:35 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/authentication_pam.so
-rwxr-xr-x 1 mysql mysql  1595720 Sep 28 19:35 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/firewall.so
-rwxr-xr-x 1 mysql mysql  3748543 Sep 28 19:35 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/keyring_okv.so
-rwxr-xr-x 1 mysql mysql  2283844 Sep 28 19:35 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/openssl_udf.so
-rwxr-xr-x 1 mysql mysql   567032 Sep 28 19:34 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/lib/plugin/thread_pool.so

So basically all MySQL Enterprise Server feature files.

The most imporant MySQL binaries

Let us have a look at the most important MySQL binaries:
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db
shell> ll mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade

-rwxr-xr-x 1 mysql mysql  10884339 Sep 28 20:04 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql
-rwxr-xr-x 1 mysql mysql   9815101 Sep 28 19:38 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql
-rwxr-xr-x 1 mysql mysql  11780342 Sep 28 20:06 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog
-rwxr-xr-x 1 mysql mysql  10711774 Sep 28 19:39 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog
-rwxr-xr-x 1 mysql mysql 253303409 Sep 28 20:11 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
-rwxr-xr-x 1 mysql mysql 253876847 Sep 28 19:42 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
-rwxr-xr-x 1 mysql mysql   9989115 Sep 28 20:06 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump
-rwxr-xr-x 1 mysql mysql   8921087 Sep 28 19:39 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump
-rwxr-xr-x 1 mysql mysql  10711017 Sep 28 20:04 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db
-rwxr-xr-x 1 mysql mysql   8883445 Sep 28 19:38 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db
-rwxr-xr-x 1 mysql mysql  13025173 Sep 28 20:07 mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade
-rwxr-xr-x 1 mysql mysql  11961246 Sep 28 19:40 mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade

Here you can see some differences I cannot explain. Possibly they come from the use of the different SSL and libedit/readline libraries?

The files are basicaly of the same type/style:

shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql
shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog
shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump
shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db
shell> file mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade

mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqld: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade:          ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped

When we look into the symbol tables of those 6 binaries we can see some differences which IMHO are mostly caused because of the 2 different set of libraries:

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql | cut -b20- >community.mysql
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql | cut -b20- >enterprise.mysql
diff community.mysql enterprise.mysql | less

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog | cut -b20- >community.mysqlbinlog
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqlbinlog | cut -b20- >enterprise.mysqlbinlog
diff community.mysqlbinlog enterprise.mysqlbinlog | less

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld | cut -b20- >community.mysqld
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqld | cut -b20- >enterprise.mysqld
diff community.mysqld enterprise.mysqld | less

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump | cut -b20- >community.mysqldump
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysqldump | cut -b20- >enterprise.mysqldump
diff community.mysqldump enterprise.mysqldump | less

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db | cut -b20- >community.mysql_install_db
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_install_db | cut -b20- >enterprise.mysql_install_db
diff community.mysql_install_db enterprise.mysql_install_db | less

nm mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade | cut -b20- >community.mysql_upgrade
nm mysql-advanced-5.7.16-linux-glibc2.5-x86_64/bin/mysql_upgrade | cut -b20- >enterprise.mysql_upgrade
diff community.mysql_upgrade enterprise.mysql_upgrade | less

Here are the files for your own research: mysql_community_and_enterprise_symbols.tar.gz

Some more details:

shell> grep zlib community.mysql enterprise.mysql
community.mysql:zlibCompileFlags
community.mysql:zlibVersion
enterprise.mysql:COMP_zlib
enterprise.mysql:COMP_zlib_cleanup
enterprise.mysql:zlibCompileFlags
enterprise.mysql:zlib_method_nozlib
enterprise.mysql:zlibVersion

shell> diff community.mysqld enterprise.mysqld  | grep -i partit
no differences

shell> diff community.mysqld enterprise.mysqld  | grep -i err_                 
< err_helper
> ERR_add_error_data
> ERR_add_error_vdata
> ERR_clear_error
> err_defaults
> ERR_error_string
> ERR_error_string_n
> err_fns
> ERR_free_strings
> ERR_func_error_string
> ERR_get_error
> ERR_get_error_line
> ERR_get_error_line_data
> ERR_get_err_state_table
> ERR_get_implementation
> ERR_get_next_error_library
> ERR_get_state
> ERR_get_string_table
> ERR_lib_error_string
> ERR_load_ASN1_strings
> ERR_load_BIO_strings
> ERR_load_BN_strings
> ERR_load_BUF_strings
> ERR_load_CMS_strings
> ERR_load_COMP_strings
> ERR_load_CONF_strings
> ERR_load_crypto_strings
> ERR_load_CRYPTO_strings
> ERR_load_DH_strings
> ERR_load_DSA_strings
> ERR_load_DSO_strings
> ERR_load_ECDH_strings
> ERR_load_ECDSA_strings
> ERR_load_EC_strings
> ERR_load_ENGINE_strings
> ERR_load_ERR_strings
> ERR_load_EVP_strings
> ERR_load_OBJ_strings
> ERR_load_OCSP_strings
> ERR_load_PEM_strings
> ERR_load_PKCS12_strings
> ERR_load_PKCS7_strings
> ERR_load_RAND_strings
> ERR_load_RSA_strings
> ERR_load_SSL_strings
> ERR_load_strings
> ERR_load_TS_strings
> ERR_load_UI_strings
> ERR_load_X509_strings
> ERR_load_X509V3_strings
> ERR_peek_error
> ERR_peek_error_line
> ERR_peek_error_line_data
> ERR_peek_last_error
> ERR_peek_last_error_line
> ERR_peek_last_error_line_data
> ERR_pop_to_mark
> ERR_print_errors
> ERR_print_errors_cb
> ERR_print_errors_fp
> ERR_put_error
> ERR_reason_error_string
> ERR_release_err_state_table
> ERR_remove_state
> ERR_remove_thread_state
> ERR_set_error_data
> ERR_set_implementation
> ERR_set_mark
> err_state_LHASH_COMP
> err_state_LHASH_HASH
> ERR_str_functs
> err_string_data_LHASH_COMP
> err_string_data_LHASH_HASH
> ERR_str_libraries
> ERR_str_reasons
> ERR_unload_strings
> int_err_del
> int_err_del_item
> int_err_get
> int_err_get_item
> int_err_get_next_lib
> int_err_library_number
> int_err_set_item
< yaERR_error_string
< yaERR_error_string_n
< yaERR_free_strings
< yaERR_get_error
< yaERR_get_error_line_data
< yaERR_GET_REASON
< yaERR_peek_error
< yaERR_print_errors_fp
< yaERR_remove_state
< _ZZ18yaERR_error_stringE3msg

MySQL Enterprise Server and MySQL Community Server packages

This is not directly MySQL Server related but it affects operation as well: We found today that the MySQL Enterprise Server RPM package and the MySQL Community Server packages were not prepared the same. The MySQL Enterprise Server for MySQL 5.7.15 package was still using sysV init scripts whereas the MySQL 5.7.15 Community Server package seems to use already SystermD unit files (since MySQL 5.7.6: Managing MySQL Server with systemd). This has change from MySQL 5.7.15 to MySQL 5.7.16 Enterprise Server. So if you side-grade from MySQL Community to MySQL Enterprise Server you might experience some surprises...

Multi-Instance set-up with MySQL Enterprise Server 5.7 on RHEL 7 with SystemD

$
0
0

In our current project the customer wants to install and run multiple MySQL Enterprise Server 5.7 Instances on the same machine (yes, I know about virtualization (we run on kvm), containers, Docker, etc.). He wants to use Red Hat Enterprise Linux (RHEL) 7 which brings the additional challenge of SystemD. So mysqld_multi is NOT an option any more.

We studied the MySQL documentation about the topic: Configuring Multiple MySQL Instances Using systemd. But to be honest: It was not really clear to me how to do the job...

So we started to work out our own cook-book which I want to share here.

The requirements are as follows:

  • Only ONE version of MySQL Enterprise Server binaries at a time is available. If you want to have more complicated set-ups (multi version) consider our MyEnv.
  • Because Segregation of Duties is an issue for this customer from the financial industries we are not allowed to use the operating system root user or have sudo privileges.
  • We have to work with the operating system user mysql as non privileged user.

Preparation work for the operating system administrator

This is the only work which has to be done under a privileged account (root):

shell> sudo yum install libaio
shell> sudo groupadd mysql
shell> sudo useradd -r -g mysql -s /bin/bash mysql
shell> sudo cp mysqld@.service /etc/systemd/system/

Installation of MySQL Enterprise Server binaries as non privileged user

To perform this task we need the generic MySQL Binary Tar Balls which you can get from the Oracle Software Delivery Cloud:

shell> mkdir /home/mysql/product
shell> cd /home/mysql/product
shell> tar xf /download/mysql-<version>.tar.gz
shell> ln -s mysql-<version> mysql-5.7.x
shell> ln -s mysql-5.7.x mysql
shell> echo 'export PATH=$PATH:/home/mysql/product/mysql/bin'>> ~/.bashrc
shell> . ~/.bashrc

Creating, Starting and Stopping several MySQL Enterprise Server Instances

shell> export INSTANCE_NAME=TMYSQL01   # and TMYSQL02 and TMYSQL03
shell> mkdir -p /mysql/${INSTANCE_NAME}/etc /mysql/${INSTANCE_NAME}/log /mysql/${INSTANCE_NAME}/data /mysql/${INSTANCE_NAME}/binlog
shell> cat /mysql/${INSTANCE_NAME}/etc/my.cnf
#
# /mysql/${INSTANCE_NAME}/etc/my.cnf
#
[mysqld]
datadir   = /mysql/${INSTANCE_NAME}/data
pid_file  = /var/run/mysqld/mysqld_${INSTANCE_NAME}.pid
log_error = /mysql/${INSTANCE_NAME}/log/error_${INSTANCE_NAME}.log
port      = 3306   # and 3307 and 3308
socket    = /var/run/mysqld/mysqld_${INSTANCE_NAME}.sock
_EOF
shell> cd /home/mysql/product/mysql
shell> bin/mysqld --defaults-file=/mysql/${INSTANCE_NAME}/etc/my.cnf --initialize --user=mysql --basedir=/home/mysql/product/mysql
shell> bin/mysqld --defaults-file=/mysql/${INSTANCE_NAME}/etc/my.cnf --daemonize >/dev/null 2>&1 &
shell> mysqladmin --user=root --socket=/var/run/mysqld/mysqld_${INSTANCE_NAME}.sock --password shutdown

So far so good. We can do everything with the database without root privileges. One thing is missing: The MySQL Database Instances should be started automatically at system reboot. For this we need a SystemD unit file:

#
# /etc/systemd/system/mysqld@.service
#

[Unit]

Description=Multi-Instance MySQL Enterprise Server
After=network.target syslog.target


[Install]

WantedBy=multi-user.target
 

[Service]

User=mysql
Group=mysql
Type=forking
PIDFile=/var/run/mysqld/mysqld_%i.pid
TimeoutStartSec=3
TimeoutStopSec=3
# true is needed for the ExecStartPre
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/run/mysqld
ExecStartPre=/bin/chown mysql: /var/run/mysqld
ExecStart=/home/mysql/product/mysql/bin/mysqld --defaults-file=/mysql/%i/etc/my.cnf --daemonize
LimitNOFILE=8192
Restart=on-failure
RestartPreventExitStatus=1
PrivateTmp=false

This file must be copied as root to:

shell> cp mysqld@.service /etc/systemd/system/

Now you can check if SystemD behaves correctly as follows:

shell> sudo systemctl daemon-reload
shell> sudo systemctl enable mysqld@TMYSQL01   # also TMYSQL02 and TMYSQL03
shell> sudo systemctl start mysqld@TMYSQL01
shell> sudo systemctl status 'mysqld@TMYSQL*'
shell> sudo systemctl start mysqld@TMYSQL01

How to go even further

If you need a more convenient or a more flexible solution you can go with our MySQL Enterprise Environment MyEnv.

Viewing all 318 articles
Browse latest View live