You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 05:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Apr 23, 2026 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3030099564,34cyclictest11475-21ssh23:50:132
3028999532,38cyclictest15682-21ssh22:40:151
30272995324,24cyclictest131rcu_preempt23:50:130
3028999487,24cyclictest844-21munin-node22:10:141
3028999484,7cyclictest131rcu_preempt00:10:131
30326994712,28cyclictest35966irq/58-eth021:15:143
3028999464,29cyclictest28840-21ssh21:35:111
3027299452,34cyclictest32353-21sh21:55:130
30326994328,10cyclictest35966irq/58-eth022:00:133
30289994328,10cyclictest0-21swapper/122:45:131
30272994314,19cyclictest12-21ksoftirqd/020:15:120
30272994314,19cyclictest12-21ksoftirqd/020:15:120
3028999423,34cyclictest25954-21ssh23:40:131
3027299421,5cyclictest131rcu_preempt20:25:130
30326994123,12cyclictest0-21swapper/322:35:153
30326994112,19cyclictest35966irq/58-eth022:15:143
3028999412,35cyclictest20475-21ssh23:55:131
3027299413,33cyclictest13503-21ssh23:15:130
3032699403,29cyclictest35966irq/58-eth021:10:133
3030099402,28cyclictest6686-21ssh22:35:142
30289993926,6cyclictest21-21ksoftirqd/122:20:131
3028999392,34cyclictest6086-21ssh00:05:131
3028999392,34cyclictest6086-21ssh00:05:131
3028999381,27cyclictest0-21swapper/121:40:131
30272993821,8cyclictest12-21ksoftirqd/023:10:120
3030099372,16cyclictest27-21ksoftirqd/222:00:132
3027299371,26cyclictest0-21swapper/021:05:140
3030099362,28cyclictest9645-21ssh23:48:152
3032699343,20cyclictest0-21swapper/300:05:133
3032699343,20cyclictest0-21swapper/300:05:133
3032699341,22cyclictest35966irq/58-eth000:25:143
30326993319,7cyclictest35966irq/58-eth000:20:133
3032699331,17cyclictest35966irq/58-eth000:00:133
3030099334,20cyclictest0-21swapper/219:10:112
3030099334,20cyclictest0-21swapper/219:10:112
3028999334,26cyclictest21-21ksoftirqd/121:05:131
3028999333,22cyclictest22173-21sshd19:35:141
3032699327,15cyclictest35966irq/58-eth023:30:133
3032699323,16cyclictest35966irq/58-eth023:50:113
3030099325,10cyclictest25478-21ssh21:15:142
3030099322,24cyclictest29516-21ssh00:00:152
30289993212,14cyclictest21-21ksoftirqd/120:10:131
30272993226,3cyclictest131rcu_preempt21:25:120
3027299322,24cyclictest17871-21ssh22:05:130
3030099312,17cyclictest27346irq/40-dwc2_hso23:05:132
3028999314,15cyclictest19098-21ssh23:00:121
3028999312,22cyclictest32344-21ssh21:55:131
30326993021,5cyclictest35966irq/58-eth022:50:133
3030099301,20cyclictest7965-21ssh23:30:132
3028999301,21cyclictest28081-21ssh23:05:141
3027299304,22cyclictest16510-21ssh21:10:130
30272993014,8cyclictest30265-21cyclictest23:20:130
3032699295,13cyclictest35966irq/58-eth023:35:123
3030099298,10cyclictest31437-21sh23:25:132
3030099293,15cyclictest27346irq/40-dwc2_hso19:55:142
3027299298,9cyclictest131rcu_preempt20:00:130
3027299295,18cyclictest12-21ksoftirqd/019:35:130
3027299291,22cyclictest10169-21ssh22:55:140
30272992911,3cyclictest131rcu_preempt22:20:120
3030099283,18cyclictest27346irq/40-dwc2_hso20:05:132
3030099283,18cyclictest27346irq/40-dwc2_hso19:25:122
3030099282,19cyclictest0-21swapper/220:25:132
3030099282,11cyclictest6202-21ssh00:05:152
3030099282,11cyclictest6202-21ssh00:05:152
3030099282,11cyclictest20586-21ssh23:55:152
3027299284,15cyclictest0-21swapper/019:30:140
3027299282,19cyclictest31557-21ssh23:25:150
3032699273,18cyclictest0-21swapper/323:25:153
30326992718,6cyclictest0-21swapper/323:45:303
3030099272,20cyclictest27346irq/40-dwc2_hso23:40:142
3030099271,17cyclictest27346irq/40-dwc2_hso20:50:122
3028999278,13cyclictest30265-21cyclictest19:55:141
3028999271,18cyclictest10942-21ssh21:25:121
3027299272,20cyclictest31390-21ssh23:41:140
3027299271,5cyclictest131rcu_preempt00:15:120
3032699261,16cyclictest35966irq/58-eth021:55:133
3032699260,17cyclictest35966irq/58-eth023:10:123
3030099265,12cyclictest27346irq/40-dwc2_hso20:55:132
3030099263,20cyclictest0-21swapper/221:50:132
3030099262,7cyclictest28866-21ssh21:35:122
3030099261,17cyclictest30123-21find22:30:122
3030099261,16cyclictest20093-21cut20:40:162
3030099261,15cyclictest21271-21ssh22:25:142
3028999262,18cyclictest1163-21ssh22:50:131
3027299263,3cyclictest131rcu_preempt23:30:320
3027299263,15cyclictest27666-21ssh00:35:140
3027299262,18cyclictest19873-21ssh21:30:140
3030099253,14cyclictest27346irq/40-dwc2_hso20:00:132
3030099253,12cyclictest27346irq/40-dwc2_hso23:15:132
3030099251,17cyclictest27346irq/40-dwc2_hso00:25:122
3027299252,16cyclictest29498-21ssh00:00:140
30272992516,5cyclictest12-21ksoftirqd/022:35:140
30272992514,4cyclictest12-21ksoftirqd/023:55:130
30326992416,5cyclictest0-21swapper/300:10:153
3032699241,14cyclictest35966irq/58-eth020:55:133
30300992416,5cyclictest930-21snmpd23:10:152
30300992412,9cyclictest5296-21ssh21:40:062
3030099241,18cyclictest1988-21diskmemload21:20:122
3028999242,17cyclictest16116-21diskmemload23:30:151
3028999241,6cyclictest53646usb-storage19:10:111
3028999241,6cyclictest53646usb-storage19:10:101
3028999241,18cyclictest22476-21rm23:20:141
3027299242,3cyclictest131rcu_preempt00:30:130
3027299242,19cyclictest8967-21ssh22:00:140
30272992417,4cyclictest0-21swapper/021:15:150
30326992310,9cyclictest35966irq/58-eth021:45:123
3030099232,6cyclictest0-21swapper/223:36:162
3030099232,16cyclictest602-21ssh00:20:142
3030099232,16cyclictest27346irq/40-dwc2_hso22:45:122
3030099232,11cyclictest9655-21grep19:20:192
30300992311,5cyclictest17784-21ssh22:05:062
3028999233,11cyclictest211ksoftirqd/119:45:121
3028999232,16cyclictest28211-21apt-get20:15:121
3028999232,16cyclictest28211-21apt-get20:15:121
3028999231,19cyclictest1795-21ls21:20:021
3027299232,16cyclictest16116-21diskmemload22:15:140
30272992315,5cyclictest0-21swapper/022:50:020
3032699229,5cyclictest0-21swapper/320:15:123
3032699229,5cyclictest0-21swapper/320:15:123
3032699225,12cyclictest0-21swapper/323:55:133
30326992214,3cyclictest0-21swapper/321:40:133
3032699221,13cyclictest35966irq/58-eth022:55:133
3030099225,3cyclictest0-21swapper/200:10:152
3030099225,15cyclictest5328-21grep22:50:292
3030099225,10cyclictest10161-21rm22:55:122
3030099224,16cyclictest19152-21rm21:10:222
3030099222,11cyclictest27346irq/40-dwc2_hso22:10:132
3030099221,8cyclictest1968366ssh22:40:312
30300992216,3cyclictest5718-21ssh22:15:212
30300992215,5cyclictest17302-21ssh21:45:212
3028999226,11cyclictest21-21ksoftirqd/123:28:151
3028999222,14cyclictest3389-21ssh22:15:141
3028999222,14cyclictest17073-21diskstats23:35:161
3028999221,9cyclictest8355-21uname23:10:261
3027299224,10cyclictest844-21munin-node00:25:140
3027299222,14cyclictest2623-21diskstats23:45:150
3027299221,6cyclictest131rcu_preempt22:10:130
3027299221,3cyclictest131rcu_preempt21:23:420
3027299221,2cyclictest131rcu_preempt23:00:130
3027299221,18cyclictest13573-21ls20:30:230
3027299221,17cyclictest16936-21ssh23:35:120
3032699215,9cyclictest0-21swapper/322:45:133
3032699213,12cyclictest0-21swapper/323:40:143
3032699212,12cyclictest0-21swapper/322:10:133
30326992110,7cyclictest0-21swapper/321:25:133
3030099218,6cyclictest8650-21cat22:00:022
3030099217,11cyclictest24901-21cut20:45:152
3030099212,14cyclictest22170-21apt-get19:35:132
3030099212,10cyclictest18216-21kthreadcore20:35:222
30300992113,4cyclictest19056-21/usr/sbin/munin23:00:132
3030099211,10cyclictest1240-21ps19:45:202
30300992110,5cyclictest2182-21ssh00:38:052
3028999212,13cyclictest844-21munin-node19:40:141
3028999211,16cyclictest27651-21ssh00:35:131
3028999211,15cyclictest1989-21sh21:20:111
3028999211,12cyclictest12358-21latency_hist19:25:021
3027299213,3cyclictest131rcu_preempt20:50:130
3027299213,15cyclictest3611-21kthreadcore20:55:190
3027299211,3cyclictest131rcu_preempt22:53:560
3027299211,3cyclictest131rcu_preempt00:07:440
3027299211,3cyclictest131rcu_preempt00:07:430
3027299211,2cyclictest131rcu_preempt21:35:140
30272992111,8cyclictest0-21swapper/000:14:320
3027299211,17cyclictest28521-21ls20:45:280
3027299211,17cyclictest18722-21cut20:05:160
3032699209,7cyclictest35966irq/58-eth023:05:133
30326992013,5cyclictest0-21swapper/322:20:263
30326992012,5cyclictest35966irq/58-eth021:20:123
3030099205,7cyclictest20839-21cut00:30:212
3030099205,6cyclictest29582-21kthreadcore19:40:212
3030099205,13cyclictest23546-21sh21:30:262
3030099204,7cyclictest14011-21timerwakeupswit20:30:272
3030099202,8cyclictest3103-21fschecks_time19:15:162
3030099202,8cyclictest3103-21fschecks_time19:15:152
3030099202,11cyclictest13312-21ssh22:20:192
3030099202,10cyclictest7960-21kthreadcore21:00:202
3030099201,9cyclictest30483-21sshd00:17:252
3030099201,9cyclictest18866-21kthreadcore19:30:202
3030099201,11cyclictest4498-21head20:20:272
3030099201,11cyclictest11069-21diskstats21:25:152
3030099201,10cyclictest6280-21kthreadcore19:50:212
3030099201,10cyclictest26009-21missed_timers23:20:242
3028999203,14cyclictest17284-21awk23:15:271
3028999202,13cyclictest844-21munin-node19:15:131
3028999202,13cyclictest844-21munin-node19:15:131
3028999202,11cyclictest0-21swapper/121:00:131
3028999202,10cyclictest0-21swapper/122:57:211
30289992014,4cyclictest0-21swapper/100:00:301
30289992012,4cyclictest21-21ksoftirqd/100:15:181
30289992011,7cyclictest21377-21sshd22:25:161
3028999201,15cyclictest23359-21apt-get21:50:121
3028999201,13cyclictest12724-21cut19:25:151
3028999200,15cyclictest13479-21ls20:30:221
3027299207,10cyclictest30352-21fschecks_time22:30:160
3027299202,3cyclictest131rcu_preempt21:50:130
3027299202,2cyclictest131rcu_preempt21:48:050
30272992014,4cyclictest0-21swapper/019:45:270
3027299201,11cyclictest4988-21kthreadcore19:50:180
3032699195,6cyclictest0-21swapper/322:05:133
3032699192,10cyclictest0-21swapper/320:35:043
30326991910,7cyclictest0-21swapper/321:50:163
30326991910,6cyclictest0-21swapper/323:22:193
30326991910,4cyclictest0-21swapper/323:15:263
3030099194,12cyclictest23488-21fschecks_count20:10:162
3030099194,11cyclictest0-21swapper/221:05:162
3030099191,9cyclictest28322-21fschecks_count20:15:152
3030099191,9cyclictest28322-21fschecks_count20:15:152
3030099191,10cyclictest30330-21cat19:10:022
3028999192,3cyclictest24547-21munin-run20:45:021
3028999191,5cyclictest53646usb-storage20:35:051
30289991911,6cyclictest167-21systemd-network23:49:561
3028999191,14cyclictest21584-21ssh22:05:261
3027299198,8cyclictest20049-21df_inode20:40:150
3027299197,10cyclictest17129-21cron19:30:010
3027299192,4cyclictest53646usb-storage19:20:040
3027299192,14cyclictest31754-21latency_hist19:45:010
3027299191,9cyclictest131rcu_preempt19:15:150
3027299191,9cyclictest131rcu_preempt19:15:150
30272991912,4cyclictest536-21usb-storage19:10:050
30272991912,4cyclictest536-21usb-storage19:10:050
30272991912,4cyclictest15633-21/usr/sbin/munin22:40:190
3027299191,16cyclictest785-21fschecks_time20:20:160
3027299191,11cyclictest584-21ssh00:20:130
3027299191,10cyclictest480-21vmstat23:05:320
3032699189,7cyclictest0-21swapper/322:30:393
3032699189,7cyclictest0-21swapper/321:36:373
3032699189,6cyclictest35966irq/58-eth000:35:133
3032699182,10cyclictest0-21swapper/320:25:133
3032699182,10cyclictest0-21swapper/320:00:153
30326991812,4cyclictest35966irq/58-eth022:25:143
30326991810,6cyclictest0-21swapper/321:31:233
3028999188,8cyclictest16291-21cut21:45:201
3028999183,9cyclictest0-21swapper/123:52:231
3028999182,12cyclictest2035-21diskstats20:55:151
3028999181,9cyclictest8101-21kthreadcore20:25:201
3028999181,7cyclictest1778-21grep00:20:201
30289991811,5cyclictest16622-21ssh21:10:141
3028999181,14cyclictest7788-21cat19:50:281
3028999181,14cyclictest30150-21ssh22:30:131
3028999181,14cyclictest19863-21ssh21:30:131
3028999181,12cyclictest8966-21sh22:00:131
3027299182,4cyclictest131rcu_preempt22:25:150
3027299181,3cyclictest131rcu_preempt21:40:140
30272991810,5cyclictest8199-21kthreadcore21:00:210
3032699178,6cyclictest0-21swapper/300:15:073
3032699174,8cyclictest0-21swapper/319:35:143
3032699172,10cyclictest33-21ksoftirqd/300:30:123
3032699172,10cyclictest0-21swapper/320:50:133
3032699171,9cyclictest0-21swapper/319:55:133
30326991711,4cyclictest0-21swapper/323:00:153
3028999179,6cyclictest9763-21forks00:25:161
3028999179,5cyclictest10276-21ntp_192.168.11422:35:261
3028999172,10cyclictest0-21swapper/120:20:141
3028999170,15cyclictest13964-21cut20:00:181
3027299172,10cyclictest0-21swapper/020:10:140
3027299171,4cyclictest131rcu_preempt19:55:200
3027299171,13cyclictest15108-21apt-get20:35:120
3032699168,5cyclictest35966irq/58-eth022:40:153
3032699162,9cyclictest0-21swapper/319:10:053
3032699162,9cyclictest0-21swapper/319:10:053
3032699161,9cyclictest33-21ksoftirqd/319:50:133
3032699161,9cyclictest0-21swapper/320:45:133
3032699161,12cyclictest33-21ksoftirqd/319:20:253
3028999165,6cyclictest0-21swapper/120:05:141
3028999163,10cyclictest16116-21diskmemload00:30:131
3028999162,10cyclictest0-21swapper/120:45:131
3028999162,10cyclictest0-21swapper/119:30:141
30289991611,3cyclictest29970-21cut20:50:161
3032699152,9cyclictest0-21swapper/321:05:143
3032699151,8cyclictest0-21swapper/319:15:223
3032699151,8cyclictest0-21swapper/319:15:223
3028999151,11cyclictest30327-21apt-key19:10:011
3027299152,11cyclictest30280-21touch19:10:020
3032699147,5cyclictest35966irq/58-eth020:30:193
3032699143,8cyclictest0-21swapper/320:20:163
3032699141,7cyclictest0-21swapper/321:00:153
3032699141,10cyclictest0-21swapper/319:45:123
3032699140,11cyclictest33-21ksoftirqd/319:42:033
3032699138,3cyclictest33-21ksoftirqd/319:33:193
3032699132,8cyclictest0-21swapper/320:45:013
3032699122,6cyclictest0-21swapper/320:10:143
3032699122,6cyclictest0-21swapper/319:25:143
3032699121,8cyclictest0-21swapper/320:05:153
3032699111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional