You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 04:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Jul 14, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
354349122430,8sleep20-21swapper/219:05:272
354409822260,8sleep126-21ksoftirqd/119:10:001
354320327240,25sleep00-21swapper/019:05:230
354376726240,15sleep30-21swapper/319:06:143
35577992560,2sleep13056-21PK-Backend19:45:131
35888862510,4sleep20-21swapper/221:15:012
3544240992924,3cyclictest3593972-21sh21:30:000
3544245992613,4cyclictest3564205-21dump-pmu-power20:04:591
3544250992522,2cyclictest3575817-21strings20:35:173
3544248992512,11cyclictest3578386-21unin-run20:45:012
3544248992511,12cyclictest3601152-21cron21:50:012
354424099248,14cyclictest3636370-21turbostat23:30:010
139099240,14rtkit-daemon0-21swapper/222:51:062
139099240,14rtkit-daemon0-21swapper/220:00:442
36086872230,2sleep13056-21PK-Backend22:10:131
35876622230,5sleep33056-21PK-Backend21:10:133
354424899233,18cyclictest3655462-21cron00:25:002
3544248992320,2cyclictest0-21swapper/221:27:232
354424899231,9cyclictest36331452/usr/sbin/munin23:20:182
354424599230,21cyclictest3056-21PK-Backend22:45:351
354424099234,17cyclictest3580759-21/usr/sbin/munin20:50:160
36572582220,3sleep20-21swapper/200:30:002
36391162220,5sleep30-21swapper/323:35:203
3544250992220,1cyclictest0-21swapper/323:08:553
3544250992220,1cyclictest0-21swapper/322:57:043
3544250992220,1cyclictest0-21swapper/320:43:013
3544250992212,7cyclictest3622482-21cron22:50:003
354425099220,21cyclictest411rcuc/319:55:013
354425099220,20cyclictest3056-21PK-Backend20:34:093
3544248992220,1cyclictest0-21swapper/223:50:542
3544248992220,1cyclictest0-21swapper/220:18:162
3544248992220,1cyclictest0-21swapper/219:59:362
3544245992220,1cyclictest3056-21PK-Backend21:38:081
354424099228,6cyclictest436-20systemd-journal22:35:010
354425099219,11cyclictest42-21ksoftirqd/300:00:013
3544250992119,1cyclictest3056-21PK-Backend19:40:083
3544250992119,1cyclictest0-21swapper/321:41:543
3544250992119,1cyclictest0-21swapper/321:04:433
3544250992119,1cyclictest0-21swapper/320:46:593
3544250992118,2cyclictest0-21swapper/300:09:373
3544250992118,2cyclictest0-21swapper/300:09:373
3544250992113,5cyclictest3562488-21munin-run20:00:003
3544250992112,5cyclictest436-20systemd-journal23:45:003
354425099210,2cyclictest0-21swapper/300:10:113
354425099210,20cyclictest0-21swapper/322:09:263
354425099210,20cyclictest0-21swapper/321:25:183
354425099210,19cyclictest3056-21PK-Backend21:55:333
354425099210,19cyclictest3056-21PK-Backend21:54:593
354425099210,19cyclictest3056-21PK-Backend19:22:503
3544248992119,1cyclictest0-21swapper/223:55:402
3544248992119,1cyclictest0-21swapper/223:10:262
3544248992119,1cyclictest0-21swapper/222:46:452
3544248992119,1cyclictest0-21swapper/222:43:012
3544248992119,1cyclictest0-21swapper/222:24:282
3544248992119,1cyclictest0-21swapper/222:08:132
3544248992119,1cyclictest0-21swapper/221:51:292
3544248992119,1cyclictest0-21swapper/219:51:322
3544248992119,1cyclictest0-21swapper/219:48:552
3544248992119,1cyclictest0-21swapper/219:31:242
3544248992119,1cyclictest0-21swapper/219:14:352
3544248992119,1cyclictest0-21swapper/200:32:592
3544248992117,3cyclictest0-21swapper/223:15:182
3544248992115,4cyclictest3581813-21cron20:55:012
3544248992114,5cyclictest3550083-21cron19:25:002
3544248992113,6cyclictest3582266-21grep20:55:122
354424899210,20cyclictest0-21swapper/223:31:412
354424899210,20cyclictest0-21swapper/222:30:162
354424899210,20cyclictest0-21swapper/221:24:462
354424899210,20cyclictest0-21swapper/221:06:002
354424899210,20cyclictest0-21swapper/220:06:462
3544245992119,1cyclictest3056-21PK-Backend00:09:501
3544245992119,1cyclictest3056-21PK-Backend00:09:501
3544245992119,1cyclictest0-21swapper/119:26:411
3544245992117,3cyclictest3612847-21ntp_states22:20:201
3544245992115,4cyclictest3567932-21munin-run20:14:591
3544245992112,7cyclictest3568211-21sshd20:15:131
354424599211,16cyclictest3572551-21seq20:25:231
354424599210,1cyclictest0-21swapper/119:17:211
3544240992119,1cyclictest0-21swapper/023:13:590
3544240992116,3cyclictest3548345-21dump-pmu-power19:19:590
354424099211,1cyclictest0-21swapper/022:57:510
354424099210,20cyclictest0-21swapper/023:43:540
3544250992018,1cyclictest3056-21PK-Backend22:37:583
3544250992018,1cyclictest3056-21PK-Backend22:21:563
3544250992018,1cyclictest3056-21PK-Backend19:25:063
3544250992018,1cyclictest0-21swapper/322:50:423
3544250992018,1cyclictest0-21swapper/321:34:033
3544250992018,1cyclictest0-21swapper/320:50:233
3544250992018,1cyclictest0-21swapper/319:47:103
3544250992017,2cyclictest3056-21PK-Backend21:17:413
3544250992015,3cyclictest3647919-21python300:00:233
3544250992013,5cyclictest3566238-21wc20:10:013
3544250992012,6cyclictest3609891-21readlink22:15:003
354425099201,17cyclictest3597099-21python321:35:223
354425099200,1cyclictest3056-21PK-Backend21:06:133
354425099200,1cyclictest3056-21PK-Backend20:15:083
354425099200,19cyclictest3056-21PK-Backend22:30:423
354425099200,19cyclictest3056-21PK-Backend00:24:123
354425099200,19cyclictest0-21swapper/323:31:393
354425099200,19cyclictest0-21swapper/323:00:503
354425099200,19cyclictest0-21swapper/322:40:353
354425099200,19cyclictest0-21swapper/321:24:443
354425099200,19cyclictest0-21swapper/320:55:303
354425099200,19cyclictest0-21swapper/320:03:403
354425099200,19cyclictest0-21swapper/319:10:073
3544248992018,1cyclictest0-21swapper/222:58:012
3544248992018,1cyclictest0-21swapper/222:36:392
3544248992018,1cyclictest0-21swapper/222:27:392
3544248992018,1cyclictest0-21swapper/221:37:172
3544248992018,1cyclictest0-21swapper/219:41:182
3544248992018,1cyclictest0-21swapper/219:15:432
3544248992014,4cyclictest978-21snmpd00:02:252
3544248992013,5cyclictest3643064-21sshd23:47:132
354424899200,19cyclictest0-21swapper/222:17:112
354424899200,19cyclictest0-21swapper/221:34:062
354424899200,19cyclictest0-21swapper/221:15:072
354424899200,19cyclictest0-21swapper/219:36:172
3544245992018,1cyclictest3056-21PK-Backend22:06:201
3544245992018,1cyclictest0-21swapper/121:07:321
3544245992014,4cyclictest3574869-21cron20:35:001
3544245992012,6cyclictest3617742-21cut22:35:121
354424599200,18cyclictest3648280-21taskset00:02:121
354424599200,18cyclictest3056-21PK-Backend20:48:401
3544240992018,1cyclictest0-21swapper/021:21:380
3544240992018,1cyclictest0-21swapper/020:49:410
3544240992018,1cyclictest0-21swapper/020:02:030
3544240992018,1cyclictest0-21swapper/000:24:440
3544240992018,1cyclictest0-21swapper/000:13:000
3544240992016,2cyclictest3555818-21apt19:40:120
354424099200,1cyclictest0-21swapper/022:20:390
354424099200,1cyclictest0-21swapper/021:33:220
354424099200,1cyclictest0-21swapper/020:21:360
354424099200,19cyclictest0-21swapper/023:01:360
354424099200,19cyclictest0-21swapper/021:11:040
354424099200,19cyclictest0-21swapper/020:36:510
354424099200,19cyclictest0-21swapper/019:45:120
354424099200,19cyclictest0-21swapper/019:13:080
354424099200,19cyclictest0-21swapper/000:26:250
139099200,14rtkit-daemon0-21swapper/300:38:193
139099200,14rtkit-daemon0-21swapper/223:06:442
139099200,14rtkit-daemon0-21swapper/220:45:172
139099200,13rtkit-daemon0-21swapper/221:58:352
3544250991917,1cyclictest3056-21PK-Backend19:34:133
3544250991917,1cyclictest3056-21PK-Backend19:15:193
3544250991917,1cyclictest0-21swapper/323:47:403
3544250991917,1cyclictest0-21swapper/323:15:303
3544250991917,1cyclictest0-21swapper/321:45:053
3544250991914,3cyclictest3643569-21apt23:50:113
3544250991912,5cyclictest3658045-21cat00:30:163
3544250991911,5cyclictest0-21swapper/323:10:183
354425099190,1cyclictest3056-21PK-Backend22:26:023
354425099190,1cyclictest3056-21PK-Backend20:24:553
354425099190,1cyclictest3056-21PK-Backend20:14:353
354425099190,1cyclictest0-21swapper/320:25:293
354425099190,18cyclictest0-21swapper/322:01:563
354424899190,1cyclictest0-21swapper/222:02:452
354424599198,8cyclictest3655460-21cron00:25:001
3544245991917,1cyclictest3056-21PK-Backend23:34:211
3544245991917,1cyclictest3056-21PK-Backend23:29:211
3544245991917,1cyclictest3056-21PK-Backend21:32:271
3544245991917,1cyclictest0-21swapper/123:22:531
3544245991917,1cyclictest0-21swapper/123:17:181
3544245991917,1cyclictest0-21swapper/121:17:261
3544245991916,2cyclictest3056-21PK-Backend21:45:191
3544245991913,4cyclictest3545072-21strings19:10:161
3544245991912,5cyclictest3553324-21sed19:30:221
3544245991912,5cyclictest3550087-21apt19:25:001
354424599190,17cyclictest3056-21PK-Backend23:56:041
3544240991917,1cyclictest3056-21PK-Backend22:12:280
3544240991917,1cyclictest0-21swapper/023:05:550
3544240991917,1cyclictest0-21swapper/022:42:290
3544240991917,1cyclictest0-21swapper/022:00:390
3544240991917,1cyclictest0-21swapper/021:19:350
3544240991917,1cyclictest0-21swapper/021:05:100
3544240991917,1cyclictest0-21swapper/020:16:420
3544240991917,1cyclictest0-21swapper/019:57:440
3544240991917,1cyclictest0-21swapper/019:21:470
3544240991912,6cyclictest3611633-21seq22:17:380
3544240991912,5cyclictest3552423-21cat19:30:130
354424099190,1cyclictest0-21swapper/022:07:350
354424099190,1cyclictest0-21swapper/020:44:340
354424099190,1cyclictest0-21swapper/020:31:230
354424099190,1cyclictest0-21swapper/020:07:300
354424099190,18cyclictest0-21swapper/023:56:460
354424099190,18cyclictest0-21swapper/023:22:380
354424099190,18cyclictest0-21swapper/022:51:280
354424099190,18cyclictest0-21swapper/021:53:420
354424099190,18cyclictest0-21swapper/020:27:380
354424099190,18cyclictest0-21swapper/019:38:100
139099190,13rtkit-daemon0-21swapper/323:25:463
139099190,13rtkit-daemon0-21swapper/220:12:292
36331762180,2sleep33056-21PK-Backend23:20:123
3544250991816,1cyclictest3056-21PK-Backend00:26:283
354425099181,15cyclictest3652233-21/usr/sbin/munin00:15:133
354425099180,1cyclictest0-21swapper/322:15:483
3544248991812,5cyclictest3584400-21cat21:00:172
3544248991811,5cyclictest3641358-21seq23:42:322
3544248991810,6cyclictest436-20systemd-journal00:16:592
3544248991810,6cyclictest436-20systemd-journal00:09:552
3544248991810,6cyclictest436-20systemd-journal00:09:552
3544245991816,1cyclictest0-21swapper/120:23:471
3544245991815,2cyclictest3056-21PK-Backend20:43:051
3544245991812,3cyclictest3660687-21idleruntime-cro00:40:001
3544240991816,1cyclictest0-21swapper/023:46:440
3544240991816,1cyclictest0-21swapper/023:15:070
3544240991816,1cyclictest0-21swapper/020:12:240
3544240991816,1cyclictest0-21swapper/019:53:580
3544240991811,5cyclictest3649229-21awk00:05:180
3544240991811,5cyclictest3649229-21awk00:05:180
3544240991811,5cyclictest3613976-21grep22:25:130
139099180,12rtkit-daemon0-21swapper/220:34:512
35986102170,4sleep13056-21PK-Backend21:40:171
3544250991715,1cyclictest3056-21PK-Backend19:38:393
354424899179,6cyclictest883-21dbus-daemon23:05:002
3544248991715,1cyclictest0-21swapper/223:38:172
3544248991711,5cyclictest3570348-21basename20:20:162
3544248991710,6cyclictest3634904-21apt23:25:122
3544248991710,5cyclictest3659788-21memory00:35:172
354424599179,5cyclictest3615745-21grep22:30:121
354424599172,10cyclictest3601432-21apt21:50:121
3544245991712,3cyclictest3603655-21kernelversion21:55:181
3544245991712,3cyclictest3587266-21acpi21:10:091
3544245991711,5cyclictest3554486-21which19:35:151
3544245991711,4cyclictest3615314-21ls22:30:001
354424599171,12cyclictest3625964-21turbostat22:59:591
3544240991715,1cyclictest0-21swapper/022:37:150
3544240991715,1cyclictest0-21swapper/021:41:120
3544240991715,1cyclictest0-21swapper/021:38:360
3544240991715,1cyclictest0-21swapper/021:02:120
3544240991715,1cyclictest0-21swapper/000:15:170
3544240991710,6cyclictest3600424-21awk21:45:180
354424099170,15cyclictest3648519-21turbostat00:04:590
35928362160,3chrt0-21swapper/121:25:151
354424899169,5cyclictest3598672-21ntpq21:40:192
3544248991614,1cyclictest0-21swapper/220:35:252
3544248991611,4cyclictest28650irq/123-ahci[0022:15:002
354424899161,11cyclictest3650517-21apt00:10:112
3544248991610,5cyclictest3572349-21grep20:25:182
354424599169,5cyclictest3657859-21forks00:30:141
354424599169,5cyclictest3590719-21date21:20:011
354424599169,4cyclictest3656796-21timerwakeupswit00:25:241
3544245991614,1cyclictest3056-21PK-Backend00:18:011
354424599160,1cyclictest3056-21PK-Backend23:40:501
354424099168,6cyclictest3551804-21cron19:29:590
354424099162,5cyclictest13-21ksoftirqd/000:38:120
3544240991614,1cyclictest0-21swapper/023:37:520
3544240991614,1cyclictest0-21swapper/023:32:590
3544240991611,3cyclictest3658943-21idleruntime-cro00:35:000
3544240991611,3cyclictest3621692-21grep22:45:160
354424599158,4cyclictest3555807-21acpi19:40:111
354424599157,6cyclictest3575523-21seq20:35:121
3544245991513,1cyclictest3056-21PK-Backend22:53:191
3544245991513,1cyclictest3056-21PK-Backend22:40:471
3544245991513,1cyclictest3056-21PK-Backend20:56:321
3544245991513,1cyclictest3056-21PK-Backend19:54:191
3544245991513,1cyclictest0-21swapper/121:04:501
3544245991512,2cyclictest3056-21PK-Backend19:55:481
354424599151,11cyclictest3611851-21awk22:20:011
354424599150,14cyclictest3056-21PK-Backend23:48:281
354424599150,13cyclictest3056-21PK-Backend23:00:071
354424099158,5cyclictest3603370-21cat21:55:140
354424099152,4cyclictest13-21ksoftirqd/020:58:120
354424099150,13cyclictest3643546-21sort23:50:100
354424899149,3cyclictest3550518-21cpuspeed_turbos19:25:122
354424599148,5cyclictest3630392-21awk23:10:171
3544245991412,1cyclictest3056-21PK-Backend00:10:251
3544245991411,2cyclictest3056-21PK-Backend23:54:021
3544245991411,2cyclictest3056-21PK-Backend20:08:111
3544245991410,3cyclictest3056-21PK-Backend23:37:141
354424599140,1cyclictest3056-21PK-Backend22:03:221
354424599139,3cyclictest3056-21PK-Backend20:50:181
3544245991210,1cyclictest3056-21PK-Backend23:06:561
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional