You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 13:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Sun May 10, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31388321290,14sleep33056-21PK-Backend20:50:143
39353421270,3sleep33056-21PK-Backend00:35:213
27598728060,13sleep10-21swapper/119:05:091
27742727441,26sleep30-21swapper/319:09:353
27744926344,13sleep00-21swapper/019:09:510
27711326343,14sleep20-21swapper/219:05:312
3404232610,4sleep00-21swapper/022:05:170
3007162580,3sleep00-21swapper/020:10:250
3832212430,3sleep30-21swapper/300:05:223
3244092330,3sleep20-21swapper/221:20:162
2986942330,4chrt3056-21PK-Backend20:05:211
3883662310,3sleep10-21swapper/100:20:241
27764599309,19cyclictest306334-21unin-run20:30:001
277645992812,14cyclictest384657-21ntp_kernel_err00:10:171
277645992812,14cyclictest384657-21ntp_kernel_err00:10:171
277651992710,6cyclictest2638-21gdbus22:01:523
27764799270,25cyclictest332997-21diskstats21:45:142
27764599271,24cyclictest323730-21grep21:20:001
2961572260,3sleep10-21swapper/120:00:111
27764599269,15cyclictest340129-21expr22:05:131
27765199252,19cyclictest371806-21apt23:35:113
277647992516,7cyclictest2270-21gdbus23:25:032
27764599256,17cyclictest338141-21ls22:00:121
27764599252,19cyclictest0-21swapper/120:55:131
277645992519,5cyclictest0-21swapper/120:54:021
27764599251,6cyclictest317000-21/usr/sbin/munin21:00:201
277651992415,7cyclictest354814-21python322:45:213
277651992412,10cyclictest292538-21df_inode19:50:133
27764599248,14cyclictest319365-21sed21:05:181
277645992411,11cyclictest313243-21latency_hist20:50:011
27764599240,19cyclictest3056-21PK-Backend19:31:351
27764599240,19cyclictest3056-21PK-Backend19:31:351
277647992316,6cyclictest1-21systemd21:05:012
27764599237,14cyclictest320683-21cpuspeed_turbos21:10:131
27764599237,14cyclictest309210-21sed20:35:201
27764599236,16cyclictest394212-21date00:40:011
27764599236,15cyclictest348483-21sh22:30:001
27764599236,15cyclictest299798-21cpuspeed_turbos20:10:111
27764599232,18cyclictest0-21swapper/120:40:111
27764599230,20cyclictest344770-21awk22:20:001
27764599230,18cyclictest3056-21PK-Backend21:28:431
277651992220,1cyclictest0-21swapper/323:45:043
277651992220,1cyclictest0-21swapper/321:41:133
277651992220,1cyclictest0-21swapper/319:47:513
277647992220,1cyclictest0-21swapper/223:18:222
277647992214,6cyclictest367799-21apt-get23:25:002
277645992220,1cyclictest3056-21PK-Backend00:06:471
277645992219,2cyclictest3056-21PK-Backend22:49:061
277645992219,2cyclictest3056-21PK-Backend22:31:221
277645992217,4cyclictest3056-21PK-Backend23:01:031
27764599221,19cyclictest386551-21python300:15:211
27764599220,21cyclictest3056-21PK-Backend23:47:331
27764599220,20cyclictest3056-21PK-Backend22:44:271
27764599220,20cyclictest3056-21PK-Backend22:13:511
277641992220,1cyclictest0-21swapper/000:10:110
277641992220,1cyclictest0-21swapper/000:10:110
139099220,4rtkit-daemon1389-21rtkit-daemon21:24:570
27765199212,5cyclictest381236-21ntp_states00:00:203
277651992119,1cyclictest3056-21PK-Backend00:29:113
277651992119,1cyclictest0-21swapper/323:52:343
277651992119,1cyclictest0-21swapper/323:31:523
277651992119,1cyclictest0-21swapper/323:09:483
277651992119,1cyclictest0-21swapper/322:58:403
277651992119,1cyclictest0-21swapper/322:34:473
277651992119,1cyclictest0-21swapper/322:06:353
277651992119,1cyclictest0-21swapper/320:15:423
277651992119,1cyclictest0-21swapper/319:59:343
277651992115,4cyclictest336716-21/usr/sbin/munin21:55:133
27765199210,8cyclictest365551-21awk23:15:223
27765199210,20cyclictest3056-21PK-Backend20:26:373
27765199210,20cyclictest0-21swapper/323:01:303
27765199210,19cyclictest341927-21df_inode22:10:143
27765199210,19cyclictest3056-21PK-Backend21:07:473
27765199210,19cyclictest3056-21PK-Backend19:12:153
277647992119,1cyclictest0-21swapper/222:09:142
277647992119,1cyclictest0-21swapper/221:38:392
277647992119,1cyclictest0-21swapper/220:17:472
277647992119,1cyclictest0-21swapper/219:44:572
277647992114,5cyclictest436-20systemd-journal19:20:012
277647992114,5cyclictest286630-21sshd19:34:242
277647992114,5cyclictest286630-21sshd19:34:242
27764799210,20cyclictest370741-21strings23:30:192
27764799210,20cyclictest0-21swapper/222:00:502
27764799210,20cyclictest0-21swapper/221:54:132
27764799210,20cyclictest0-21swapper/219:48:542
27764799210,20cyclictest0-21swapper/219:20:492
27764799210,1cyclictest0-21swapper/220:28:422
27764599217,12cyclictest3056-21PK-Backend22:50:541
27764599217,12cyclictest0-21swapper/119:10:281
27764599213,3cyclictest391376-21uname00:30:161
277645992119,1cyclictest3056-21PK-Backend23:59:301
277645992119,1cyclictest3056-21PK-Backend23:41:051
277645992119,1cyclictest3056-21PK-Backend19:23:031
277645992119,1cyclictest0-21swapper/120:23:541
277645992119,1cyclictest0-21swapper/120:23:541
27764599211,4cyclictest306711-21expr20:30:121
27764599211,16cyclictest3056-21PK-Backend22:55:131
27764599210,2cyclictest0-21swapper/119:40:191
27764599210,20cyclictest3056-21PK-Backend23:23:411
27764599210,20cyclictest3056-21PK-Backend19:48:331
27764599210,20cyclictest3056-21PK-Backend00:03:081
27764599210,19cyclictest3056-21PK-Backend21:40:051
277641992119,1cyclictest0-21swapper/020:07:580
277641992113,6cyclictest314528-21sed20:50:220
27764199210,20cyclictest0-21swapper/023:11:310
27764199210,20cyclictest0-21swapper/021:09:020
277651992018,1cyclictest3056-21PK-Backend21:12:203
277651992018,1cyclictest0-21swapper/322:26:563
277651992018,1cyclictest0-21swapper/321:30:123
277651992018,1cyclictest0-21swapper/319:22:093
277651992017,2cyclictest3056-21PK-Backend00:15:393
277651992017,2cyclictest3056-21PK-Backend00:10:113
277651992017,2cyclictest3056-21PK-Backend00:10:113
277651992017,2cyclictest0-21swapper/323:55:243
27765199201,5cyclictest283970-21irqstats19:25:163
277651992015,4cyclictest978-21snmpd19:18:003
277651992015,3cyclictest978-21snmpd22:23:233
27765199201,4cyclictest353080-21python322:40:193
277651992013,5cyclictest330403-21sshd21:37:243
27765199200,1cyclictest0-21swapper/321:51:003
27765199200,19cyclictest3056-21PK-Backend20:58:083
27765199200,19cyclictest0-21swapper/322:15:273
277647992018,1cyclictest0-21swapper/200:25:282
27764799200,1cyclictest0-21swapper/220:56:552
27764799200,1cyclictest0-21swapper/200:20:482
27764799200,1cyclictest0-21swapper/200:08:382
27764799200,19cyclictest0-21swapper/223:51:012
27764799200,19cyclictest0-21swapper/222:46:462
27764799200,19cyclictest0-21swapper/220:38:032
27764799200,19cyclictest0-21swapper/219:39:552
27764799200,19cyclictest0-21swapper/200:36:372
27764799200,19cyclictest0-21swapper/200:31:572
27764799200,19cyclictest0-21swapper/200:16:082
27764799200,19cyclictest0-21swapper/200:11:292
27764799200,19cyclictest0-21swapper/200:11:292
27764599203,2cyclictest0-21swapper/121:20:231
277645992018,1cyclictest3056-21PK-Backend23:52:131
277645992018,1cyclictest3056-21PK-Backend23:27:191
277645992018,1cyclictest3056-21PK-Backend23:15:361
277645992018,1cyclictest0-21swapper/123:37:131
277645992018,1cyclictest0-21swapper/100:25:111
27764599200,8cyclictest361870-21/usr/sbin/munin23:05:191
27764599200,7cyclictest0-21swapper/121:50:201
27764599200,7cyclictest0-21swapper/119:40:001
27764599200,1cyclictest3056-21PK-Backend19:56:161
27764599200,19cyclictest3056-21PK-Backend19:54:401
277641992018,1cyclictest0-21swapper/020:42:260
277641992013,6cyclictest305555-21ntp_kernel_pll_20:25:190
277641992011,7cyclictest317888-21python321:00:210
2826282190,5sleep00-21swapper/019:20:200
277651991917,1cyclictest3056-21PK-Backend20:45:033
277651991917,1cyclictest3056-21PK-Backend20:43:273
277651991917,1cyclictest3056-21PK-Backend00:34:383
277651991917,1cyclictest0-21swapper/323:26:253
277651991917,1cyclictest0-21swapper/323:20:573
277651991917,1cyclictest0-21swapper/322:54:003
277651991912,6cyclictest324730-21awk21:20:203
27765199191,16cyclictest2156-21gnome-shell19:33:093
27765199191,16cyclictest2156-21gnome-shell19:33:093
27765199190,1cyclictest3056-21PK-Backend21:01:053
27765199190,1cyclictest3056-21PK-Backend20:36:313
27765199190,1cyclictest3056-21PK-Backend20:31:103
27765199190,1cyclictest3056-21PK-Backend20:09:553
27765199190,1cyclictest3056-21PK-Backend00:23:433
27765199190,1cyclictest0-21swapper/320:01:573
27765199190,1cyclictest0-21swapper/319:41:153
27765199190,17cyclictest303387-21cstates20:20:123
27765199190,17cyclictest303387-21cstates20:20:123
277647991915,3cyclictest357821-21tr22:55:132
277647991914,3cyclictest978-21snmpd20:03:592
277647991911,6cyclictest376041-21awk23:45:192
27764799191,14cyclictest336315-21taskset21:55:042
27764799190,1cyclictest0-21swapper/223:10:042
27764799190,1cyclictest0-21swapper/222:33:142
27764799190,1cyclictest0-21swapper/222:21:452
27764799190,1cyclictest0-21swapper/221:30:272
27764799190,1cyclictest0-21swapper/220:54:312
27764799190,18cyclictest0-21swapper/222:19:552
27764799190,18cyclictest0-21swapper/221:06:002
27764799190,18cyclictest0-21swapper/219:14:412
277645991917,1cyclictest3056-21PK-Backend19:15:331
277645991916,2cyclictest3056-21PK-Backend22:20:071
27764599190,8cyclictest337840-21sleep22:00:001
27764599190,1cyclictest3056-21PK-Backend23:10:091
27764599190,1cyclictest3056-21PK-Backend21:30:181
277641991915,3cyclictest311969-21cstates20:45:130
277641991914,4cyclictest309574-21sshd20:37:430
277641991912,6cyclictest388500-21awk00:20:240
277641991912,6cyclictest291515-21sshd19:45:230
277641991911,6cyclictest286299-21users19:30:220
277641991911,6cyclictest286299-21users19:30:210
277651991816,1cyclictest0-21swapper/323:42:133
277651991813,3cyclictest326432-21ntp_states21:25:173
277651991811,5cyclictest334118-21pluginstate21:50:013
27765199180,6cyclictest363813-21threads23:10:203
27765199180,16cyclictest351065-21memory22:35:163
277647991816,1cyclictest0-21swapper/220:42:012
277647991816,1cyclictest0-21swapper/220:05:372
277647991815,2cyclictest1895000-21snapd23:08:282
277647991812,4cyclictest360501-21sshd23:00:242
277647991811,6cyclictest380494-21apt00:00:122
277647991811,6cyclictest331941-21sed21:40:232
277647991811,5cyclictest346478-21cron22:25:032
277647991811,5cyclictest326197-21cut21:25:142
277647991811,5cyclictest312279-21cut20:45:172
277647991810,6cyclictest436-20systemd-journal22:40:002
277647991810,6cyclictest352220-21apt22:40:112
27764599185,11cyclictest3056-21PK-Backend19:29:451
277645991816,1cyclictest3056-21PK-Backend23:30:571
277645991813,4cyclictest3056-21PK-Backend20:19:211
27764599180,2cyclictest3056-21PK-Backend21:45:211
27764199183,14cyclictest1095-21runrttasks21:15:570
277641991816,1cyclictest0-21swapper/022:21:150
277641991816,1cyclictest0-21swapper/022:15:210
277641991816,1cyclictest0-21swapper/020:16:230
277641991816,1cyclictest0-21swapper/020:02:310
277641991812,5cyclictest13-21ksoftirqd/022:01:520
277641991812,5cyclictest13-21ksoftirqd/000:20:010
277641991811,5cyclictest320873-21cat21:10:150
27764199181,12cyclictest389164-21apt00:25:130
27765199172,13cyclictest299800-21tr20:10:113
27764799179,6cyclictest292914-21ntp_states19:50:182
27764799172,14cyclictest1095-21runrttasks19:57:262
277647991715,1cyclictest0-21swapper/223:40:402
277647991715,1cyclictest0-21swapper/223:36:482
277647991715,1cyclictest0-21swapper/222:12:522
277647991713,3cyclictest978-21snmpd20:13:472
277647991713,2cyclictest320681-21tr21:10:132
277647991711,5cyclictest356046-21cat22:50:102
27764799171,14cyclictest883-21dbus-daemon20:21:522
27764799171,14cyclictest883-21dbus-daemon20:21:522
277647991711,4cyclictest2156-21gnome-shell23:58:082
277647991710,6cyclictest283901-21basename19:25:152
277647991710,5cyclictest322972-21ntpq21:15:202
277645991714,2cyclictest3056-21PK-Backend22:36:361
277641991715,1cyclictest0-21swapper/021:57:490
277641991713,2cyclictest394099-21latency_hist00:40:000
277641991712,4cyclictest355535-21mktemp22:50:010
277641991712,4cyclictest13-21ksoftirqd/019:38:030
277641991712,3cyclictest303251-21cpuspeed_turbos20:20:120
277641991712,3cyclictest303251-21cpuspeed_turbos20:20:120
277641991711,5cyclictest883-21dbus-daemon23:58:030
277641991711,4cyclictest381533-21grep00:00:230
277641991710,5cyclictest367799-21apt-get23:24:590
27764199170,16cyclictest0-21swapper/019:27:360
27765199160,14cyclictest286935-21/usr/sbin/munin19:35:223
277647991614,1cyclictest3056-21PK-Backend20:31:542
27764199169,5cyclictest368720-21cat23:25:150
277641991614,1cyclictest3056-21PK-Backend21:51:340
277641991614,1cyclictest0-21swapper/000:08:360
277641991612,3cyclictest362615-21ls23:10:000
277641991612,3cyclictest348484-21sh22:30:000
277641991612,3cyclictest315768-21interrupts20:55:170
277641991612,2cyclictest334090-21idleruntime-cro21:50:000
277641991611,4cyclictest327920-21sed21:30:170
277641991610,5cyclictest372249-21cat23:35:150
277641991610,5cyclictest356922-21grep22:50:190
277641991610,5cyclictest307808-21seq20:32:180
277641991610,5cyclictest13-21ksoftirqd/019:40:110
277641991610,4cyclictest2156-21gnome-shell22:43:090
27764199160,6cyclictest279467-21sort19:15:010
27764199160,3cyclictest141rcu_preempt21:25:520
27764599151,4cyclictest330008-21chrt21:35:191
27764199159,5cyclictest390878-21find00:30:110
27764199159,5cyclictest359541-21grep23:00:120
277641991514,1cyclictest0-21swapper/019:58:250
277641991512,2cyclictest373707-21tr23:40:110
277641991511,3cyclictest375245-21/usr/sbin/munin23:45:180
277641991511,2cyclictest13-21ksoftirqd/022:55:130
277641991510,4cyclictest350622-21tr22:35:110
277641991510,3cyclictest378465-21latency_hist23:55:000
27764199149,4cyclictest371109-21wc23:30:230
27764199148,5cyclictest341732-21tr22:10:130
27764199147,5cyclictest330820-21latency_hist21:40:010
277641991412,1cyclictest0-21swapper/019:15:470
27764199141,11cyclictest365587-21grep23:15:220
27764199140,3cyclictest141rcu_preempt21:43:450
277651991311,1cyclictest3056-21PK-Backend21:17:123
27764199138,4cyclictest349431-21cut22:30:170
27764199138,3cyclictest292711-21latency19:50:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional