You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-05 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Jan 04, 2026 12:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52687621090,9sleep234-21ksoftirqd/207:05:202
52666528038,35sleep10-21swapper/107:05:151
5747062770,10sleep3574711-21cpuspeed_turbos09:20:133
52741127739,14sleep00-21swapper/007:05:460
6145582750,8sleep14497-21PK-Backend11:10:311
5694882750,4sleep30-21swapper/309:05:213
6316702720,7sleep0161rcuc/012:00:270
6169332680,6sleep30-21swapper/311:20:023
527869996722,21cyclictest632348-21cron12:05:001
527877996617,19cyclictest438-20systemd-journal08:23:293
527877996312,17cyclictest576783-21kernelversion09:25:233
52786599632,36cyclictest544066-21taskset07:53:000
52786599631,37cyclictest575911-21taskset09:24:080
527877996213,31cyclictest539796-21irqstats07:40:213
527877996110,35cyclictest2069-21gnome-shell07:32:083
52744926141,14sleep30-21swapper/307:06:183
197399610,9rtkit-daemon1972-21rtkit-daemon09:42:360
5851862600,5chrt0-21swapper/209:50:142
52786599602,34cyclictest593061-21taskset10:12:090
52786599601,35cyclictest565084-21taskset08:52:200
52786599601,35cyclictest565084-21taskset08:52:200
197399600,9rtkit-daemon1972-21rtkit-daemon10:55:031
197399600,9rtkit-daemon1972-21rtkit-daemon10:26:251
527877995917,24cyclictest619151-21df_inode11:25:163
527877995917,24cyclictest619151-21df_inode11:25:163
527877995915,25cyclictest588756-21diskstats10:00:173
527877995915,24cyclictest593341-21munin-run10:15:013
52786599592,32cyclictest588874-21kernelversion10:00:230
52786599591,13cyclictest618287-21taskset11:22:190
52786599581,34cyclictest538641-21taskset07:35:420
527877995710,22cyclictest621254-21ntp_states11:30:253
197399570,9rtkit-daemon1972-21rtkit-daemon08:38:171
197399570,8rtkit-daemon1972-21rtkit-daemon09:55:341
527877995612,20cyclictest590394-21expr10:05:143
527877995611,20cyclictest631302-21hddtemp_smartct12:00:203
197399560,8rtkit-daemon1972-21rtkit-daemon12:36:301
527877995510,20cyclictest566423-21/usr/sbin/munin08:55:263
527865995529,22cyclictest577710-21cron09:30:000
197399550,9rtkit-daemon1972-21rtkit-daemon12:27:561
527877995416,10cyclictest595078-40lsb_release10:17:053
527877995415,17cyclictest549866-21cmp08:10:023
527877995414,17cyclictest637076-21sed12:15:333
527877995414,17cyclictest637076-21sed12:15:333
527877995414,17cyclictest541970-21grep07:45:303
527877995411,17cyclictest538401-21python307:35:283
52786599541,48cyclictest590321-21expr10:05:140
197399540,8rtkit-daemon1972-21rtkit-daemon08:49:591
527877995316,9cyclictest643718-21hddtemp_smartct12:35:193
527877995315,15cyclictest543490-21awk07:50:223
527877995314,16cyclictest636011-21cat12:15:023
527877995311,18cyclictest638190-21df_inode12:20:153
527877995311,15cyclictest0-21swapper/307:57:063
527865995320,28cyclictest619515-21cat11:25:240
527865995320,28cyclictest619515-21cat11:25:240
52787799527,25cyclictest1054-21snmpd08:40:083
527877995221,10cyclictest641024-21apt-get12:30:003
527877995211,18cyclictest642775-40lsb_release12:32:023
527877995211,17cyclictest583773-21memory09:45:223
527877995211,17cyclictest582544-21users09:40:323
527877995211,16cyclictest629754-21memory11:55:243
197399520,8rtkit-daemon1972-21rtkit-daemon08:10:331
197399520,8rtkit-daemon1972-21rtkit-daemon07:55:561
52787799519,15cyclictest564225-21diskstats08:50:173
52787799519,15cyclictest564225-21diskstats08:50:173
527877995116,31cyclictest562867-21sh08:45:263
527877995111,18cyclictest587171-21grep09:55:223
527877995111,17cyclictest628016-21memory11:50:223
527877995111,17cyclictest628016-21memory11:50:223
527877995111,17cyclictest568050-21ntp_states09:00:253
527877995111,16cyclictest634096-21latency_hist12:10:023
527865995117,29cyclictest594851-21wc10:15:310
527865995117,29cyclictest560619-21expr08:40:140
527877995014,14cyclictest558854-21wc08:35:133
527877995011,16cyclictest622986-21ntp_states11:35:263
527877995011,16cyclictest586381-21munin-run09:55:003
527877995011,15cyclictest581201-21munin-run09:40:013
527865995027,18cyclictest541630-21memory07:45:230
527865995020,26cyclictest573136-21df_inode09:15:220
527865995017,29cyclictest635404-21timerwakeupswit12:10:290
527877994912,14cyclictest617106-21sort11:20:143
527865994919,25cyclictest612345-21cut11:05:210
527865994917,27cyclictest636711-21wc12:15:260
527865994917,27cyclictest636711-21wc12:15:260
5927172480,4chrt0-21swapper/110:10:301
52787799489,16cyclictest604925-21df10:45:153
527877994818,25cyclictest529749-21cat07:15:023
52786599487,36cyclictest603804-21sort10:40:300
527865994819,25cyclictest599823-21iwlist10:30:210
527865994819,24cyclictest608742-21ntpq10:55:240
527865994816,28cyclictest556698-21grep08:30:020
52786599481,42cyclictest4497-21PK-Backend08:15:240
52786599481,11cyclictest536484-21tr07:30:220
5416312470,6sleep10-21swapper/107:45:221
52787799479,15cyclictest1-21systemd11:00:023
527877994711,13cyclictest549670-21sed08:10:013
52786599474,39cyclictest615046-21dump-pmu-power11:15:000
527865994716,27cyclictest643093-21ls12:35:010
527865994714,29cyclictest633644-21timerwakeupswit12:05:300
52787799469,16cyclictest552094-21df08:15:183
52787799468,16cyclictest626275-21sed11:45:223
527877994619,22cyclictest547816-21cat08:05:013
527877994618,23cyclictest611568-21munin-run11:05:003
527877994618,23cyclictest611568-21munin-run11:05:003
527877994611,12cyclictest1054-21snmpd09:12:403
52786599467,33cyclictest438-20systemd-journal08:15:010
527865994616,26cyclictest584510-21seq09:48:270
197399460,37rtkit-daemon2481-40tracker-miner-f12:32:021
527877994514,26cyclictest604450-21wc10:45:023
527877994513,9cyclictest573163-21fw_packets09:15:253
527865994516,25cyclictest601863-21open_files10:35:260
527865994516,25cyclictest4497-21PK-Backend07:57:400
527865994512,28cyclictest539748-21cat07:40:200
527877994418,21cyclictest534542-21pluginstate07:25:253
527877994417,23cyclictest597958-21df10:25:143
527877994415,23cyclictest601079-21acpi10:35:113
527877994414,26cyclictest612052-21cstates11:05:143
527877994411,30cyclictest0-21swapper/310:22:423
527877994411,30cyclictest0-21swapper/310:22:423
52786599449,30cyclictest0-21swapper/009:10:280
527865994418,22cyclictest558387-21latency_hist08:35:010
527865994417,23cyclictest559304-21needreboot08:35:220
527877994317,22cyclictest530852-21wc07:15:223
527877994317,22cyclictest530852-21wc07:15:223
527877994316,22cyclictest625552-21latency_hist11:45:023
52786599438,31cyclictest549431-21apt-get08:10:010
52786599432,37cyclictest610634-21python311:00:300
52786599432,37cyclictest610634-21python311:00:300
527865994316,23cyclictest639324-21grep12:25:010
52786599431,37cyclictest587524-21python309:55:300
527865994312,27cyclictest620622-21cpuspeed_turbos11:30:140
527865994311,27cyclictest644085-21open_files12:35:270
5481682420,5sleep1251rcuc/108:05:151
527877994216,22cyclictest533119-21users07:20:333
527877994215,22cyclictest1054-21snmpd10:30:163
527869994222,16cyclictest538347-21processes07:35:281
52786599428,30cyclictest553611-21cpuspeed_turbos08:20:140
527865994218,20cyclictest584890-21/usr/sbin/munin09:50:230
527865994218,20cyclictest4497-21PK-Backend10:52:070
527865994217,21cyclictest1137-21runrttasks09:30:150
52786599421,36cyclictest570060-21grep09:05:310
527865994212,26cyclictest566626-21timerwakeupswit08:55:290
197399420,35rtkit-daemon4497-21PK-Backend09:19:121
6012262410,7sleep2601231-21cpuspeed_turbos10:35:142
527877994114,6cyclictest613983-21hddtemp_smartct11:10:183
527877994110,11cyclictest2257-21gsd-housekeepin09:30:403
527869994124,12cyclictest611342-21unattended-upgr11:02:451
527869994124,12cyclictest611342-21unattended-upgr11:02:451
52786599412,34cyclictest641121-21latency_hist12:30:010
527865994119,18cyclictest597765-21tr10:25:130
52786599411,36cyclictest528948-21python307:10:270
52786599411,35cyclictest629791-21missed_timers11:55:250
52786599411,34cyclictest568926-21wc09:05:020
527877994014,22cyclictest557573-21needreboot08:30:213
527874994023,12cyclictest438-20systemd-journal07:54:292
52786599404,31cyclictest625899-21cat11:45:140
52786599402,34cyclictest623114-21python311:35:290
52786599401,35cyclictest531317-21timerwakeupswit07:15:320
52786599401,35cyclictest531317-21timerwakeupswit07:15:310
527865994011,25cyclictest627605-21tr11:50:140
527865994011,25cyclictest627605-21tr11:50:140
527877993918,17cyclictest556065-21chrt08:25:283
527874993914,13cyclictest141rcu_preempt11:43:492
527869993922,13cyclictest633713-21users12:05:311
52786599397,28cyclictest615807-21kernelversion11:15:210
52786599392,33cyclictest922-21avahi-daemon08:50:010
52786599392,33cyclictest604611-21/usr/sbin/munin10:45:130
52786599392,32cyclictest535576-21dpkg07:30:020
52786599391,33cyclictest597260-21sed10:25:030
52786599391,33cyclictest597260-21sed10:25:020
52786599391,33cyclictest580326-21sort09:35:230
5331362380,13sleep013-21ksoftirqd/007:20:340
527874993823,11cyclictest1086-21sshd12:06:292
527874993820,13cyclictest438-20systemd-journal08:46:102
52786999387,27cyclictest621611-21vmstat11:30:311
52786999381,33cyclictest4497-21PK-Backend11:06:261
52786599381,33cyclictest624525-21ls11:40:220
197399380,32rtkit-daemon0-21swapper/112:14:411
527877993711,23cyclictest0-21swapper/310:50:303
527874993713,12cyclictest141rcu_preempt12:28:042
527874993713,12cyclictest141rcu_preempt10:47:112
527874993713,12cyclictest141rcu_preempt08:59:262
52786999371,10cyclictest0-21swapper/107:44:201
527874993620,12cyclictest568181-21python309:00:292
527874993620,12cyclictest3039-21gdbus10:26:082
527874993614,18cyclictest614337-21ntp_states11:10:262
527874993613,11cyclictest141rcu_preempt10:43:082
527874993612,12cyclictest141rcu_preempt11:59:022
527874993612,10cyclictest141rcu_preempt11:25:162
527874993612,10cyclictest141rcu_preempt11:25:152
527869993618,13cyclictest590822-21ntpq10:05:231
527869993617,14cyclictest594190-21df10:15:161
527869993613,19cyclictest566672-21/usr/sbin/munin08:55:301
52786999361,10cyclictest4497-21PK-Backend11:45:101
527865993611,21cyclictest547710-21turbostat08:04:590
197399360,29rtkit-daemon1972-21rtkit-daemon12:23:011
527874993520,11cyclictest561490-21seq08:40:342
527874993519,12cyclictest628169-21cat11:50:252
527874993519,12cyclictest628169-21cat11:50:242
527874993519,12cyclictest627173-21/usr/sbin/munin11:50:022
527874993519,12cyclictest607625-21seq10:53:312
527874993519,11cyclictest622966-21ntpq11:35:252
527874993519,11cyclictest546129-21grep08:00:032
527874993514,17cyclictest548167-21seq08:05:142
527874993512,11cyclictest141rcu_preempt08:18:292
527874993511,12cyclictest141rcu_preempt10:31:342
527869993520,11cyclictest616781-21perf11:20:001
52786999351,9cyclictest0-21swapper/107:51:371
527869993518,12cyclictest438-20systemd-journal08:52:301
527869993518,12cyclictest438-20systemd-journal08:52:291
197399350,29rtkit-daemon0-21swapper/107:34:501
527874993419,11cyclictest923-21dbus-daemon07:20:002
527874993419,11cyclictest923-21dbus-daemon07:20:002
527874993419,11cyclictest532088-21awk07:20:142
527874993418,12cyclictest597057-21apt-get10:25:002
527874993418,12cyclictest597057-21apt-get10:25:002
527874993417,13cyclictest534629-21python307:25:272
527874993416,13cyclictest555555-21df08:25:152
527874993414,6cyclictest141rcu_preempt11:15:112
52787499341,12cyclictest5747082grep09:20:132
527874993411,11cyclictest141rcu_preempt11:32:592
527874993411,10cyclictest141rcu_preempt09:45:122
52786999341,9cyclictest4497-21PK-Backend11:35:281
52786999341,9cyclictest4497-21PK-Backend09:53:031
527869993419,11cyclictest589133-21ntpq10:00:261
527869993419,11cyclictest572224-21apt-get09:15:001
52786999341,8cyclictest4497-21PK-Backend07:20:141
527869993418,12cyclictest578732-21ntpq09:30:381
527869993418,12cyclictest560866-21irqstats08:40:191
527869993418,11cyclictest582926-21/usr/sbin/munin09:45:001
527869993417,13cyclictest583752-21cat09:45:211
527869993416,14cyclictest636748-21tr12:15:261
527869993416,14cyclictest636748-21tr12:15:261
52786999341,30cyclictest0-21swapper/108:17:531
52787499332,18cyclictest0-21swapper/211:05:072
527874993319,10cyclictest529195-21seq07:10:452
527874993318,11cyclictest574277-21/usr/sbin/munin09:20:012
527874993317,12cyclictest570484-21cron09:10:002
527874993317,12cyclictest557922-21cat08:30:282
527874993315,14cyclictest34-21ksoftirqd/210:02:502
52787499331,28cyclictest4497-21PK-Backend07:39:252
52786999337,22cyclictest575981-21cron09:25:011
527869993323,6cyclictest620235-21sh11:30:001
527869993323,6cyclictest620235-21sh11:30:001
527869993323,6cyclictest595783-21cpuspeed_turbos10:20:151
527869993323,6cyclictest595783-21cpuspeed_turbos10:20:151
527869993323,5cyclictest26-21ksoftirqd/110:37:141
52786999331,8cyclictest4497-21PK-Backend08:30:571
527869993318,11cyclictest553857-21awk08:20:181
52786999331,15cyclictest4497-21PK-Backend07:18:261
52786999331,15cyclictest4497-21PK-Backend07:18:251
6175972320,6chrt4497-21PK-Backend11:20:212
527874993225,4cyclictest0-21swapper/212:24:372
527874993222,6cyclictest34-21ksoftirqd/208:00:142
527874993222,5cyclictest34-21ksoftirqd/207:30:232
527874993221,6cyclictest34-21ksoftirqd/208:14:202
527874993218,10cyclictest637561-21sh12:20:002
527874993218,10cyclictest637561-21sh12:20:002
527874993218,10cyclictest609275-21sshd10:57:302
527874993217,11cyclictest641855-21expr12:30:162
527874993216,11cyclictest438-20systemd-journal09:35:292
52787499321,5cyclictest4497-21PK-Backend11:01:452
52787499321,5cyclictest4497-21PK-Backend11:01:442
52787499321,4cyclictest4497-21PK-Backend10:13:552
52787499321,28cyclictest4497-21PK-Backend12:02:432
52787499321,27cyclictest4497-21PK-Backend09:42:572
527874993210,4cyclictest141rcu_preempt07:45:292
527869993226,4cyclictest0-21swapper/111:55:131
527869993225,4cyclictest4497-21PK-Backend07:11:351
527869993218,10cyclictest534513-21uname07:25:241
52786999321,7cyclictest4497-21PK-Backend09:04:171
52786999321,4cyclictest0-21swapper/110:41:451
52786999321,28cyclictest4497-21PK-Backend09:35:451
52786999321,28cyclictest4497-21PK-Backend09:06:021
52787499315,9cyclictest141rcu_preempt10:15:382
527874993122,5cyclictest34-21ksoftirqd/208:35:292
52787499311,28cyclictest0-21swapper/209:28:522
52787499311,27cyclictest4497-21PK-Backend08:22:542
527874993110,4cyclictest141rcu_preempt09:57:532
527869993121,7cyclictest624128-21tr11:40:131
527869993119,8cyclictest555372-21tr08:25:141
527869993115,13cyclictest2094-21JS10:49:091
52786999311,28cyclictest0-21swapper/110:52:411
527869993110,5cyclictest141rcu_preempt10:49:541
52787499308,10cyclictest141rcu_preempt10:05:262
527874993023,4cyclictest4497-21PK-Backend08:54:292
527874993023,4cyclictest4497-21PK-Backend08:54:282
52787499301,8cyclictest924-21NetworkManager09:34:342
527874993015,11cyclictest570949-21expr09:10:132
527869993024,4cyclictest0-21swapper/111:53:211
527869993024,4cyclictest0-21swapper/111:53:211
52786999301,4cyclictest0-21swapper/111:21:271
527874992923,4cyclictest0-21swapper/212:38:032
527869992917,9cyclictest600823-21cron10:35:001
527869992915,10cyclictest546581-21expr08:00:141
52787499281,6cyclictest4497-21PK-Backend07:42:402
527874992716,7cyclictest634869-21cut12:10:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional