You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-18 - 03:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Mon May 18, 2026 00:44:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
61994922480,5sleep10-21swapper/119:05:141
63763821190,4sleep234-21ksoftirqd/219:55:122
6452152980,4sleep23056-21PK-Backend20:15:172
7363442970,2sleep33056-21PK-Backend00:35:143
62107828341,14sleep30-21swapper/319:09:313
62108527141,25sleep00-21swapper/019:09:360
6473242640,4sleep23056-21PK-Backend20:20:222
61964226443,14sleep20-21swapper/219:05:082
6703582500,1sleep30-21swapper/321:25:193
6814962470,3sleep00-21swapper/022:00:010
6339262450,1sleep10-21swapper/119:45:021
621287993112,17cyclictest718062-21cron23:45:003
621287993111,18cyclictest718332-21find23:45:133
62128799293,25cyclictest728756-21cron00:15:003
62128799293,25cyclictest728756-21cron00:15:003
621287992914,13cyclictest687346-21kernelversion22:15:173
6954662280,16sleep234-21ksoftirqd/222:40:012
62128799289,5cyclictest732645-21tr00:25:113
62128799288,5cyclictest727103-21date00:10:003
621287992813,13cyclictest731146-21diskstats00:20:143
6262802270,3sleep0161rcuc/019:20:220
621287992718,7cyclictest883-21dbus-daemon23:24:413
6454972260,3chrt644712-21/usr/sbin/munin20:15:201
62128799268,5cyclictest668018-21expr21:20:143
62128799267,5cyclictest725414-21wc00:04:593
62128799266,5cyclictest699068-21cat22:50:123
62128799259,5cyclictest1-21systemd23:28:013
62128799258,5cyclictest729378-21ls00:15:143
62128799258,4cyclictest653751-21expr20:40:133
62128799258,3cyclictest641501-21kernelversion20:05:183
62128799257,16cyclictest685751-21sed22:10:183
62128799257,16cyclictest685751-21sed22:10:173
62128799256,5cyclictest701651-21cat22:55:213
62128799256,4cyclictest648792-21ls20:25:163
621287992510,4cyclictest708328-21hddtemp_smartct23:15:163
62128799250,1cyclictest3056-21PK-Backend21:40:183
62128799250,19cyclictest0-21swapper/319:55:423
7263942240,2sleep00-21swapper/000:05:200
6957722240,5sleep20-21swapper/222:40:132
62128799245,5cyclictest705335-21sshd23:05:553
7085322230,3sleep20-21swapper/223:15:182
62128799238,5cyclictest632454-21cat19:40:133
62128799237,3cyclictest639826-21latency20:00:163
62128799237,3cyclictest0-21swapper/321:35:203
62128799230,22cyclictest720711-21diskstats23:50:153
62128799230,22cyclictest622615-21runrttasks19:11:153
62128799230,10cyclictest702867-21/usr/sbin/munin23:00:163
621286992315,6cyclictest640746-21cron20:05:002
621278992316,4cyclictest690056-21idleruntime-cro22:25:000
139099230,15rtkit-daemon0-21swapper/219:14:562
62128799227,4cyclictest2156-21gnome-shell21:19:093
62128799221,8cyclictest627522-21diskstats19:25:143
621287992214,6cyclictest698344-21users22:45:233
62128799221,11cyclictest659000-21tr20:55:113
62128799220,12cyclictest630590-21/usr/sbin/munin19:35:153
621286992220,1cyclictest0-21swapper/221:55:062
621286992220,1cyclictest0-21swapper/219:15:192
621286992213,7cyclictest641401-21awk20:05:172
62128699220,21cyclictest731487-21ntpq00:20:182
621278992220,1cyclictest0-21swapper/023:52:150
621278992220,1cyclictest0-21swapper/023:28:220
621278992220,1cyclictest0-21swapper/021:32:090
621278992220,1cyclictest0-21swapper/021:22:290
621278992220,1cyclictest0-21swapper/020:34:500
621278992220,1cyclictest0-21swapper/020:07:260
621278992220,1cyclictest0-21swapper/019:40:420
621278992220,1cyclictest0-21swapper/000:03:100
621278992215,5cyclictest656602-21turbostat.cron20:50:000
621278992215,4cyclictest635556-21ls19:50:000
621278992210,8cyclictest1-21systemd21:45:000
62127899220,21cyclictest0-21swapper/022:31:230
621287992118,2cyclictest3056-21PK-Backend22:28:583
621287992117,3cyclictest3056-21PK-Backend22:31:493
621287992117,3cyclictest3056-21PK-Backend21:55:253
621286992113,6cyclictest653123-21cron20:40:002
62128699210,20cyclictest0-21swapper/220:53:072
621281992113,6cyclictest643000-21awk20:10:101
621281992113,6cyclictest643000-21awk20:10:101
62128199210,3cyclictest662088-21munin-run21:05:011
62128199210,2cyclictest664262-21cnf-update-db21:10:011
62128199210,20cyclictest0-21swapper/122:30:181
62128199210,19cyclictest3056-21PK-Backend23:26:221
62128199210,19cyclictest3056-21PK-Backend20:28:581
62128199210,19cyclictest3056-21PK-Backend00:15:161
62128199210,19cyclictest3056-21PK-Backend00:10:561
62128199210,19cyclictest3056-21PK-Backend00:10:561
62127899212,14cyclictest643866-21taskset20:10:200
62127899212,14cyclictest643866-21taskset20:10:200
621278992119,1cyclictest0-21swapper/023:37:480
621278992119,1cyclictest0-21swapper/023:01:180
621278992119,1cyclictest0-21swapper/022:27:320
621278992119,1cyclictest0-21swapper/020:42:070
621278992119,1cyclictest0-21swapper/020:35:510
621278992118,2cyclictest0-21swapper/000:14:050
621278992118,2cyclictest0-21swapper/000:14:050
621278992114,6cyclictest735728-21systemctl00:35:000
62127899210,20cyclictest0-21swapper/023:55:320
62127899210,20cyclictest0-21swapper/023:14:500
62127899210,20cyclictest0-21swapper/023:14:500
62127899210,20cyclictest0-21swapper/019:56:580
62127899210,1cyclictest0-21swapper/022:48:410
62127899210,1cyclictest0-21swapper/000:17:230
139099210,14rtkit-daemon0-21swapper/221:52:032
62128799201,8cyclictest705940-21apt23:10:133
62128799201,8cyclictest705940-21apt23:10:133
621287992018,1cyclictest3056-21PK-Backend19:15:463
621287992018,1cyclictest0-21swapper/300:30:183
62128799200,9cyclictest652531-21python320:35:213
62128799200,1cyclictest3056-21PK-Backend20:20:013
62128799200,19cyclictest3056-21PK-Backend22:36:283
62128799200,19cyclictest3056-21PK-Backend22:22:293
62128799200,19cyclictest3056-21PK-Backend20:50:233
62128799200,19cyclictest3056-21PK-Backend19:49:053
62128799200,19cyclictest0-21swapper/323:55:303
62128699203,15cyclictest1095-21runrttasks20:32:152
621286992018,1cyclictest0-21swapper/223:25:162
621286992018,1cyclictest0-21swapper/222:05:122
621286992018,1cyclictest0-21swapper/221:31:532
621286992014,4cyclictest626910-21systemd-run19:25:002
621286992012,5cyclictest629030-21grep19:30:122
62128699200,1cyclictest3056-21PK-Backend00:30:252
62128699200,19cyclictest978-21snmpd19:44:392
62128699200,19cyclictest0-21swapper/223:46:042
62128699200,19cyclictest0-21swapper/222:18:312
62128699200,19cyclictest0-21swapper/222:12:562
62128699200,19cyclictest0-21swapper/222:12:562
62128699200,19cyclictest0-21swapper/219:36:362
621281992018,1cyclictest3056-21PK-Backend22:44:311
621281992012,6cyclictest736987-21sed00:35:221
62128199200,1cyclictest3056-21PK-Backend23:40:551
62128199200,1cyclictest3056-21PK-Backend22:49:111
62128199200,1cyclictest3056-21PK-Backend21:27:521
62128199200,1cyclictest0-21swapper/122:03:351
62128199200,1cyclictest0-21swapper/119:55:591
62128199200,1cyclictest0-21swapper/100:32:331
62128199200,19cyclictest3056-21PK-Backend23:37:371
62128199200,19cyclictest0-21swapper/122:10:031
62128199200,19cyclictest0-21swapper/122:10:031
62128199200,19cyclictest0-21swapper/122:09:021
62128199200,19cyclictest0-21swapper/120:40:541
62128199200,19cyclictest0-21swapper/100:20:361
62128199200,18cyclictest3056-21PK-Backend23:06:411
62128199200,18cyclictest3056-21PK-Backend22:59:311
621278992019,1cyclictest0-21swapper/023:45:390
621278992019,1cyclictest0-21swapper/021:27:430
621278992019,1cyclictest0-21swapper/019:10:270
621278992018,1cyclictest0-21swapper/022:57:260
621278992018,1cyclictest0-21swapper/022:50:430
621278992018,1cyclictest0-21swapper/022:10:550
621278992018,1cyclictest0-21swapper/022:10:550
621278992018,1cyclictest0-21swapper/020:25:440
621278992018,1cyclictest0-21swapper/020:03:470
621278992018,1cyclictest0-21swapper/019:53:410
621278992018,1cyclictest0-21swapper/000:22:570
621278992011,7cyclictest681657-21awk22:00:120
62127899200,19cyclictest0-21swapper/023:44:240
62127899200,19cyclictest0-21swapper/022:41:580
62127899200,19cyclictest0-21swapper/022:38:060
62127899200,19cyclictest0-21swapper/021:17:150
62128799191,8cyclictest715615-21ntp_states23:35:183
62128799191,8cyclictest629257-21/usr/sbin/munin19:30:133
621287991917,1cyclictest3056-21PK-Backend21:50:593
621287991917,1cyclictest0-21swapper/322:42:103
62128799191,6cyclictest714007-21python323:30:203
62128799190,1cyclictest3056-21PK-Backend22:06:073
62128799190,1cyclictest3056-21PK-Backend22:04:183
62128799190,1cyclictest3056-21PK-Backend21:47:223
62128799190,1cyclictest3056-21PK-Backend20:10:353
62128799190,1cyclictest3056-21PK-Backend20:10:353
62128799190,1cyclictest0-21swapper/321:34:033
62128799190,18cyclictest3056-21PK-Backend21:07:193
62128799190,18cyclictest3056-21PK-Backend20:48:003
62128799190,18cyclictest3056-21PK-Backend19:53:243
62128699192,11cyclictest719803-21apt-get23:50:042
621286991917,1cyclictest0-21swapper/223:11:102
621286991917,1cyclictest0-21swapper/223:11:102
621286991917,1cyclictest0-21swapper/222:59:272
621286991917,1cyclictest0-21swapper/222:54:272
621286991914,3cyclictest669236-21cron21:25:002
621286991913,4cyclictest656608-21grep20:50:002
621286991912,5cyclictest677307-21sensors_temp21:45:212
621286991911,5cyclictest715525-21grep23:35:172
621281991917,1cyclictest3056-21PK-Backend23:54:541
621281991917,1cyclictest3056-21PK-Backend20:01:331
621281991917,1cyclictest3056-21PK-Backend19:27:541
621281991917,1cyclictest0-21swapper/121:55:301
621281991917,1cyclictest0-21swapper/121:37:321
621281991913,5cyclictest626127-21ntpq19:20:191
621281991911,7cyclictest677499-21sed21:45:231
62128199190,1cyclictest3056-21PK-Backend23:31:291
62128199190,1cyclictest3056-21PK-Backend20:32:361
62128199190,1cyclictest3056-21PK-Backend19:32:471
62128199190,1cyclictest0-21swapper/122:35:591
62128199190,1cyclictest0-21swapper/122:23:351
62128199190,1cyclictest0-21swapper/121:54:291
62127899193,13cyclictest711141-21perf23:25:000
62127899192,14cyclictest707896-21chrt23:15:040
621278991917,1cyclictest0-21swapper/021:38:240
621278991917,1cyclictest0-21swapper/021:13:500
621278991913,4cyclictest684848-21sh22:10:000
62127899190,1cyclictest0-21swapper/021:05:250
62127899190,1cyclictest0-21swapper/020:24:430
62127899190,1cyclictest0-21swapper/019:18:250
62127899190,18cyclictest0-21swapper/000:26:150
139099190,13rtkit-daemon0-21swapper/223:32:572
621287991816,1cyclictest3056-21PK-Backend20:17:243
621287991811,5cyclictest42-21ksoftirqd/320:30:123
62128799180,1cyclictest0-21swapper/321:13:073
62128699186,9cyclictest34-21ksoftirqd/220:11:092
62128699186,9cyclictest34-21ksoftirqd/220:11:092
621286991816,1cyclictest0-21swapper/222:33:522
621286991816,1cyclictest0-21swapper/221:44:582
621286991816,1cyclictest0-21swapper/200:00:032
621286991814,3cyclictest978-21snmpd21:12:542
621286991812,4cyclictest727465-21cpuspeed_turbos00:10:132
621286991812,4cyclictest727465-21cpuspeed_turbos00:10:132
621286991811,5cyclictest698044-21awk22:45:202
621286991810,6cyclictest673354-21df_inode21:35:142
62128699180,1cyclictest0-21swapper/221:03:412
621281991816,1cyclictest3056-21PK-Backend23:49:341
621281991816,1cyclictest3056-21PK-Backend23:21:151
621281991816,1cyclictest3056-21PK-Backend22:54:521
621281991816,1cyclictest3056-21PK-Backend21:21:031
621281991816,1cyclictest3056-21PK-Backend20:24:321
621281991816,1cyclictest0-21swapper/119:18:081
621281991816,1cyclictest0-21swapper/119:14:151
621281991812,4cyclictest632256-21date19:40:011
621278991817,1cyclictest0-21swapper/019:26:240
621278991816,1cyclictest0-21swapper/023:07:400
621278991816,1cyclictest0-21swapper/021:51:560
621278991816,1cyclictest0-21swapper/021:45:410
621278991810,5cyclictest660039-21sshd20:56:330
62127899180,5cyclictest3056-21PK-Backend19:33:580
62128699179,6cyclictest733514-21wc00:25:222
62128699179,6cyclictest436-20systemd-journal19:50:002
62128699174,11cyclictest662912-21missed_timers21:05:182
62128699172,14cyclictest1095-21runrttasks23:59:142
62128699172,14cyclictest1095-21runrttasks23:23:262
621286991715,1cyclictest0-21swapper/222:02:492
621286991715,1cyclictest0-21swapper/221:27:212
621286991715,1cyclictest0-21swapper/220:56:392
621286991715,1cyclictest0-21swapper/219:27:562
621286991713,3cyclictest688775-21tr22:20:132
621286991710,5cyclictest703461-21sed23:00:232
621286991710,5cyclictest648212-21grep20:25:122
621286991710,5cyclictest28650irq/123-ahci[0021:15:242
621281991715,1cyclictest0-21swapper/120:59:131
621281991715,1cyclictest0-21swapper/119:52:411
621281991714,2cyclictest3056-21PK-Backend23:59:131
621281991712,3cyclictest667514-21dump-pmu-power21:20:001
621278991714,2cyclictest1095-21runrttasks21:04:030
62127899171,12cyclictest657129-21apt20:50:120
621287991612,2cyclictest626993-21latency_hist19:25:003
62128699169,5cyclictest918-21rs:main2
62128699165,10cyclictest636486-21needreboot19:50:182
62128699163,12cyclictest1095-21runrttasks00:09:252
62128699161,12cyclictest729037-21apt00:15:132
62128699161,12cyclictest436-20systemd-journal23:10:002
62128699161,11cyclictest654269-21python320:40:222
621281991614,1cyclictest3056-21PK-Backend20:35:281
621281991614,1cyclictest0-21swapper/123:15:061
621281991614,1cyclictest0-21swapper/119:49:231
621281991614,1cyclictest0-21swapper/100:27:121
621281991613,2cyclictest3056-21PK-Backend00:09:551
621281991610,5cyclictest655288-21grep20:45:131
621281991610,4cyclictest2156-21gnome-shell21:43:091
62127899168,6cyclictest883-21dbus-daemon19:40:000
621278991610,5cyclictest646153-21seq20:19:370
621281991512,2cyclictest3056-21PK-Backend21:32:181
621281991510,4cyclictest657505-21entropy20:50:151
621281991510,3cyclictest706686-21sh23:10:201
621281991510,3cyclictest706686-21sh23:10:201
621281991510,3cyclictest1095-21runrttasks22:25:161
62128199150,6cyclictest3056-21PK-Backend00:01:221
621278991510,4cyclictest687420-21sed22:15:180
621278991510,3cyclictest1095-21runrttasks23:30:170
62128799149,3cyclictest661574-21taskset21:00:223
62128699149,3cyclictest690972-21needreboot22:25:172
621286991410,3cyclictest736807-21processes00:35:192
621281991412,1cyclictest3056-21PK-Backend23:01:341
621281991412,1cyclictest3056-21PK-Backend22:15:311
62128199141,10cyclictest662327-21apt21:05:141
62127899141,10cyclictest7359412/usr/sbin/munin00:35:140
62128199122,2cyclictest141rcu_preempt20:05:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional