You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 07:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Mon Jan 05, 2026 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9128302930,7sleep04497-21PK-Backend23:40:150
81695628566,13sleep30-21swapper/319:05:463
197399830,63rtkit-daemon0-21swapper/119:05:281
197399770,5rtkit-daemon0-21swapper/022:12:450
197399760,6rtkit-daemon0-21swapper/022:48:560
197399760,5rtkit-daemon0-21swapper/021:56:130
197399750,8rtkit-daemon817419-21cyclictest20:11:080
197399750,6rtkit-daemon0-21swapper/019:16:070
197399750,5rtkit-daemon0-21swapper/019:55:550
197399750,5rtkit-daemon0-21swapper/000:10:370
81658827045,17sleep20-21swapper/219:05:252
81743199671,40cyclictest854418-21taskset20:54:132
81743199671,40cyclictest824723-21taskset19:29:342
81743199671,38cyclictest931711-21taskset00:32:172
81725626738,23sleep00-21swapper/019:09:360
81743199661,39cyclictest829922-21taskset19:44:382
81743199641,39cyclictest907005-21taskset23:24:202
81743199641,39cyclictest907005-21taskset23:24:202
81743199631,56cyclictest839842-21grep20:10:242
817431996214,42cyclictest890220-21expr22:35:152
81743199621,36cyclictest849951-21taskset20:40:172
817431996121,34cyclictest438-20systemd-journal23:27:152
81743199601,45cyclictest0-21swapper/222:17:072
81743199601,38cyclictest918811-21taskset23:55:302
81743199601,36cyclictest905119-21taskset23:17:192
81743199601,36cyclictest886026-21taskset22:22:192
81743199601,33cyclictest868894-21cat21:35:162
81743199581,51cyclictest923915-21sed00:10:302
81743199571,51cyclictest881634-21cat22:10:172
8792222560,18sleep013-21ksoftirqd/022:05:010
817431995623,29cyclictest934110-21systemd-run00:40:012
197399560,5rtkit-daemon0-21swapper/020:07:070
817431995519,31cyclictest844749-21sh20:25:162
817431995518,32cyclictest874513-21sort21:50:202
817431995518,32cyclictest874513-21sort21:50:202
817431995518,32cyclictest841327-21smartctl20:15:232
817431995517,33cyclictest842910-21expr20:20:162
81743199551,49cyclictest846882-21open_inodes20:30:272
81743199551,48cyclictest873840-21latency_hist21:50:022
81743599542,46cyclictest909479-21hddtemp_smartct23:30:183
81743199542,46cyclictest886494-21/usr/sbin/munin22:25:262
817431995422,28cyclictest866853-21/usr/sbin/munin21:30:172
817431995418,31cyclictest826694-21date19:35:022
817431995417,32cyclictest830492-21expr19:45:122
81743199541,48cyclictest899494-21wc23:00:282
81743599531,17cyclictest906096-21/usr/sbin/munin23:20:223
81743599531,17cyclictest906096-21/usr/sbin/munin23:20:223
81743199532,45cyclictest928683-21/usr/sbin/munin00:25:162
817431995316,32cyclictest870957-21cat21:40:152
81743199531,47cyclictest893458-21/usr/sbin/munin22:45:322
81743199531,47cyclictest876700-21python321:55:282
81743199531,47cyclictest836544-21grep20:00:282
81742599538,6cyclictest141rcu_preempt00:20:000
817435995225,23cyclictest850250-21needreboot20:40:243
817435995219,28cyclictest843640-21grep20:20:323
81743599521,16cyclictest862061-21/usr/sbin/munin21:15:173
81743199522,45cyclictest877935-21df_inode22:00:162
81743199522,45cyclictest856468-21/usr/sbin/munin21:00:142
817431995217,30cyclictest879473-21cpuspeed_turbos22:05:132
817431995216,32cyclictest851575-21expr20:45:132
817431995216,32cyclictest851575-21expr20:45:132
817431995216,31cyclictest902209-21cpuspeed23:10:172
817431995215,33cyclictest919800-21expr00:00:172
817431995215,32cyclictest911618-21processes23:35:262
81743199521,46cyclictest900636-21cstates23:05:152
81743199521,46cyclictest865997-21ps21:25:262
81743199521,45cyclictest892820-21systemd-udevd22:40:362
817431995213,34cyclictest916111-21cat23:50:132
81743599512,44cyclictest867316-21if_enp2s021:30:203
81743599512,12cyclictest915748-21perf23:49:593
81743599512,12cyclictest915748-21perf23:49:593
817435995117,30cyclictest923-21dbus-daemon22:20:013
817435995114,32cyclictest880918-21date22:10:013
817435995114,32cyclictest873009-21cat21:45:213
81743199512,44cyclictest928039-21sshd00:23:292
817431995118,28cyclictest913088-21latency23:40:232
817431995116,31cyclictest914384-21cat23:45:142
817431995116,31cyclictest914384-21cat23:45:142
817431995116,30cyclictest847818-21acpi20:35:112
81743199511,45cyclictest925227-21expr00:15:162
817431995114,32cyclictest858521-21expr21:05:192
817431995114,32cyclictest833914-21expr19:55:162
81743599507,8cyclictest913068-21expr23:40:233
81743599502,8cyclictest931871-21systemd-udevd00:34:163
81743599502,42cyclictest844775-21hddtemp_smartct20:25:193
817435995018,28cyclictest933369-21ntp_states00:35:273
817435995016,30cyclictest848035-21cpuspeed_turbos20:35:133
81743599501,44cyclictest893846-21df22:45:163
817435995010,11cyclictest869065-21if_err_enp2s021:35:213
81743199501,45cyclictest861931-21expr21:15:142
81743199501,45cyclictest0-21swapper/222:50:142
81743199501,44cyclictest1137-21runrttasks19:35:032
81743199501,43cyclictest860215-21expr21:10:142
817431995014,31cyclictest922024-21processes00:05:272
817431995012,33cyclictest837567-21cat20:05:142
817431995010,35cyclictest897202-21cat22:55:122
817431995010,35cyclictest897202-21cat22:55:122
81743599492,9cyclictest882433-40lsb_release22:11:003
817435994921,23cyclictest42-21ksoftirqd/320:55:293
817435994916,28cyclictest898742-21awk23:00:133
817435994914,7cyclictest856548-21cpuspeed21:00:153
81743199499,35cyclictest863734-21cat21:20:162
81743199492,43cyclictest0-21swapper/223:30:182
81743199492,42cyclictest1054-21snmpd22:30:092
8867722480,5sleep14497-21PK-Backend22:25:141
81743599482,41cyclictest858600-21df_inode21:05:213
81743599482,40cyclictest863392-21/usr/sbin/munin21:20:243
817435994814,29cyclictest825444-21hddtemp_smartct19:30:203
817435994813,12cyclictest920344-21awk00:00:313
81743599481,10cyclictest919193-21cron00:00:003
81743199481,4cyclictest0-21swapper/220:55:132
81743199481,43cyclictest1137-21runrttasks19:18:282
817431994812,32cyclictest832219-21expr19:50:192
9286082470,20sleep013-21ksoftirqd/000:25:030
81743599478,12cyclictest834442-21sh19:55:273
81743599472,41cyclictest846489-21diskstats20:30:173
817435994722,20cyclictest874464-21df21:50:163
817435994722,20cyclictest874464-21df21:50:163
817435994716,27cyclictest907324-21/usr/sbin/munin23:25:143
817435994716,26cyclictest831944-21acpi19:50:163
817435994715,28cyclictest896210-21timerandwakeup22:50:293
817435994715,28cyclictest824765-21cron19:30:013
817435994715,27cyclictest853043-21cpu20:50:133
817435994714,7cyclictest839399-21cstates20:10:173
817435994714,29cyclictest910964-21cpuspeed_turbos23:35:143
817435994714,28cyclictest884998-21cstates22:20:153
817435994714,28cyclictest870490-21dpkg21:40:033
817435994713,30cyclictest876331-21tr21:55:223
817435994713,13cyclictest903501-21seq23:13:483
817431994714,28cyclictest823243-21ls19:25:022
81743599462,39cyclictest925315-21diskstats00:15:173
817435994619,22cyclictest865120-21/usr/sbin/munin21:25:223
817435994618,25cyclictest1137-21runrttasks19:46:453
817435994616,26cyclictest1137-21runrttasks23:18:233
817435994614,28cyclictest901227-21processes23:05:283
817435994611,7cyclictest859949-21acpi21:10:113
817435994611,12cyclictest924437-21gpgv00:15:013
81743599452,9cyclictest823073-21munin-run19:25:013
817435994514,26cyclictest916214-21cstates23:50:143
81743599451,39cyclictest1054-21snmpd22:55:133
81743599451,39cyclictest1054-21snmpd22:55:123
817435994513,8cyclictest837665-21cstates20:05:153
817435994512,8cyclictest852163-21sensors_temp20:45:263
817435994512,8cyclictest852163-21sensors_temp20:45:263
817435994510,9cyclictest835766-21cpuspeed20:00:133
81743199452,39cyclictest819097-21idleruntime-cro19:15:002
197399450,31rtkit-daemon2069-21gnome-shell00:22:100
8601292440,13sleep1261ksoftirqd/121:10:141
81743599442,14cyclictest829971-21awk19:45:003
817435994410,10cyclictest826989-21seq19:35:143
817425994428,12cyclictest856212-21cron21:00:000
197399440,32rtkit-daemon4497-21PK-Backend23:28:010
8423582430,7sleep14497-21PK-Backend20:19:561
81743599438,10cyclictest877587-21cpu22:00:133
817435994317,21cyclictest841130-21cstates20:15:173
817435994313,6cyclictest928177-21latency_hist00:25:023
817435994311,7cyclictest921768-21latency00:05:223
81742599431,38cyclictest822333-21ntp_states19:20:280
81742799422,11cyclictest843393-21ntp_states20:20:261
81743599417,8cyclictest928574-21gpgv00:25:033
81743599411,14cyclictest886912-21/usr/sbin/munin22:25:193
817425994118,8cyclictest13-21ksoftirqd/023:55:260
817425994117,9cyclictest13-21ksoftirqd/021:30:140
81742599411,22cyclictest1-21systemd20:40:010
197399410,28rtkit-daemon4497-21PK-Backend23:47:121
197399410,28rtkit-daemon4497-21PK-Backend23:47:121
197399410,26rtkit-daemon0-21swapper/000:36:570
81743599401,19cyclictest4497-21PK-Backend22:40:333
817435994010,6cyclictest8209842chrt19:16:033
817427994022,13cyclictest872740-21df_inode21:45:161
817425994021,12cyclictest13-21ksoftirqd/021:43:110
81742599401,9cyclictest0-21swapper/021:25:010
197399400,25rtkit-daemon4497-21PK-Backend21:58:341
81743599391,12cyclictest819035-21chrt19:14:293
81742599396,6cyclictest141rcu_preempt20:30:490
817425993930,5cyclictest13-21ksoftirqd/000:34:580
197399390,27rtkit-daemon4497-21PK-Backend21:38:320
197399390,27rtkit-daemon4497-21PK-Backend19:52:061
81743599381,12cyclictest891340-21chrt22:38:533
817425993825,9cyclictest13-21ksoftirqd/023:05:300
197399380,26rtkit-daemon4497-21PK-Backend22:50:350
197399380,26rtkit-daemon4497-21PK-Backend21:54:041
197399380,26rtkit-daemon4497-21PK-Backend21:54:041
197399380,26rtkit-daemon4497-21PK-Backend19:43:161
197399380,25rtkit-daemon4497-21PK-Backend21:30:041
197399380,25rtkit-daemon4497-21PK-Backend20:41:531
81742799372,12cyclictest863794-21diskstats21:20:181
817427993721,11cyclictest438-20systemd-journal20:50:001
817427993721,11cyclictest438-20systemd-journal20:49:591
817427993717,15cyclictest892076-21df22:40:171
817425993725,8cyclictest13-21ksoftirqd/022:30:000
817425993720,13cyclictest827365-21grep19:35:230
81742599371,20cyclictest825199-21tr19:30:130
81742599371,20cyclictest13-21ksoftirqd/022:40:170
197399370,5rtkit-daemon0-21swapper/023:50:110
197399370,26rtkit-daemon4497-21PK-Backend22:50:091
197399370,26rtkit-daemon0-21swapper/022:59:390
197399370,26rtkit-daemon0-21swapper/022:59:380
197399370,26rtkit-daemon0-21swapper/021:09:570
197399370,25rtkit-daemon4497-21PK-Backend23:31:371
197399370,25rtkit-daemon4497-21PK-Backend20:59:091
197399370,25rtkit-daemon0-21swapper/023:02:270
197399370,25rtkit-daemon0-21swapper/020:51:220
197399370,24rtkit-daemon0-21swapper/100:26:391
81743599361,17cyclictest4497-21PK-Backend22:34:133
817427993619,13cyclictest976-21rs:main1
81742799361,14cyclictest834502-21processes19:55:291
81742599362,16cyclictest438-20systemd-journal23:49:590
81742599362,16cyclictest438-20systemd-journal23:49:590
817425993618,13cyclictest438-20systemd-journal20:25:000
81742599361,20cyclictest13-21ksoftirqd/023:40:010
197399360,25rtkit-daemon0-21swapper/021:26:150
197399360,25rtkit-daemon0-21swapper/021:11:540
197399360,24rtkit-daemon4497-21PK-Backend21:01:291
197399360,24rtkit-daemon0-21swapper/122:15:511
197399360,24rtkit-daemon0-21swapper/119:16:271
81742799351,9cyclictest4497-21PK-Backend23:20:481
81742799351,9cyclictest4497-21PK-Backend23:20:471
81742799351,9cyclictest4497-21PK-Backend21:39:261
81742799351,8cyclictest0-21swapper/100:20:441
81742799351,10cyclictest4497-21PK-Backend23:02:491
81742799351,10cyclictest4497-21PK-Backend22:45:021
817425993525,6cyclictest833437-21munin-run19:55:010
817425993519,5cyclictest13-21ksoftirqd/020:50:000
817425993519,5cyclictest13-21ksoftirqd/020:49:590
817425993514,16cyclictest438-20systemd-journal19:44:550
817425993513,6cyclictest141rcu_preempt22:38:250
817425993513,6cyclictest141rcu_preempt21:50:000
197399350,24rtkit-daemon0-21swapper/120:30:401
197399350,24rtkit-daemon0-21swapper/021:04:250
197399350,24rtkit-daemon0-21swapper/019:46:260
197399350,23rtkit-daemon0-21swapper/023:21:440
197399350,23rtkit-daemon0-21swapper/023:21:430
197399350,23rtkit-daemon0-21swapper/019:27:370
81742799349,6cyclictest8632902cat21:20:021
81742799346,8cyclictest26-21ksoftirqd/119:10:291
817427993421,9cyclictest26-21ksoftirqd/122:35:131
81742799341,9cyclictest4497-21PK-Backend23:56:381
81742799341,9cyclictest4497-21PK-Backend23:37:251
81742799341,9cyclictest4497-21PK-Backend22:06:551
81742799341,9cyclictest4497-21PK-Backend20:52:261
81742799341,8cyclictest4497-21PK-Backend20:35:081
817427993417,12cyclictest438-20systemd-journal23:40:281
81742599345,15cyclictest13-21ksoftirqd/021:50:170
81742599345,15cyclictest13-21ksoftirqd/021:50:170
81742599342,16cyclictest0-21swapper/020:25:220
817425993419,12cyclictest13-21ksoftirqd/020:17:030
817425993413,6cyclictest438-20systemd-journal22:06:290
817425993413,6cyclictest141rcu_preempt23:35:000
81742599341,17cyclictest888738-21uname22:30:200
197399340,23rtkit-daemon4497-21PK-Backend22:23:441
197399340,23rtkit-daemon0-21swapper/122:33:341
81742799331,9cyclictest4497-21PK-Backend23:29:011
81742799331,9cyclictest4497-21PK-Backend19:23:491
81742799331,9cyclictest4497-21PK-Backend00:36:461
81742799331,8cyclictest4497-21PK-Backend23:06:441
81742799331,8cyclictest4497-21PK-Backend22:58:551
81742799331,8cyclictest4497-21PK-Backend22:58:541
817427993318,11cyclictest923-21dbus-daemon20:30:011
81742799331,29cyclictest4497-21PK-Backend19:47:581
817427993312,6cyclictest141rcu_preempt00:30:141
81742599333,17cyclictest835661-21/usr/sbin/munin20:00:110
817425993322,7cyclictest919662-21tr00:00:170
817425993318,11cyclictest904058-21tr23:15:140
817425993311,6cyclictest141rcu_preempt20:45:010
81742599331,12cyclictest0-21swapper/023:10:190
197399330,5rtkit-daemon0-21swapper/022:15:170
817427993225,5cyclictest0-21swapper/100:05:201
817427993224,5cyclictest4497-21PK-Backend23:15:001
817427993216,11cyclictest438-20systemd-journal22:15:011
817425993222,6cyclictest13-21ksoftirqd/021:15:060
817425993217,11cyclictest922761-21ls00:10:010
817425993217,11cyclictest818931-21seq19:13:100
817427993124,4cyclictest4497-21PK-Backend19:29:311
817427993122,5cyclictest26-21ksoftirqd/120:10:301
817427993117,10cyclictest825533-21cut19:30:211
817427993116,11cyclictest917543-21wc23:55:021
81742799311,27cyclictest4497-21PK-Backend20:09:151
817425993118,9cyclictest886399-21cut22:25:020
9264772302,6sleep1141rcu_preempt00:20:021
817427993023,4cyclictest4497-21PK-Backend21:09:171
817427993019,7cyclictest921144-21grep00:05:011
817427993017,9cyclictest904858-21sed23:15:311
817427993017,9cyclictest826958-21tr19:35:151
81742799301,10cyclictest4497-21PK-Backend21:25:161
817427992918,8cyclictest4497-21PK-Backend20:01:161
817427992914,11cyclictest871187-21missed_timers21:40:231
81742799291,14cyclictest4497-21PK-Backend22:01:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional